Image processing device, image processing method, and image sensing apparatus

ABSTRACT

An image processing device includes: a frequency divider for performing a frequency division processing of dividing an input image into a plurality of frequency components each having a frequency band; a noise remover for performing a noise component removal processing of removing a noise component from a high frequency component in the frequency components each having the frequency band obtained by the frequency division processing by the frequency divider; an edge preservation information calculator for detecting an edge intensity based on a low frequency component in the frequency components each having the frequency band obtained by the frequency division processing by the frequency divider, and calculating edge preservation information relating to a degree of preserving an edge component based on the detected edge intensity; an edge preserving section for preserving the edge component in the high frequency component, based on the edge preservation information calculated by the edge preservation information calculator; and a frequency synthesizer for synthesizing the high frequency component whose noise component is removed by the noise remover and whose edge component is preserved by the edge preserving section, and the low frequency component, in each of the frequency bands.

This application is based on Japanese Patent Application No. 2006-185415filed on Jul. 5, 2006, the contents of which are hereby incorporated byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device capable ofremoving noise from an image, and more particularly to an imageprocessing device that enables to remove noise from a wide dynamic rangeimage obtained by an image sensing apparatus capable of performing awide dynamic range imaging, an image processing method, and an imagesensing apparatus using the image processing method.

2. Description of the Related Art

An image sensing apparatus such as a digital camera is provided with apredetermined image sensor to capture an image by an imaging operationof the image sensor. Generally, an image i.e. an image signal capturedby an image sensor includes a noise component resulting from e.g. a darkcurrent inherent to the image sensor. In response to a recent demand fora high-quality image, it is required to finely remove the noisecomponent.

As an example of the conventional noise removal methods, there is knowna method, as shown in FIG. 25, comprising: isolating a high-frequencycomponent 922 including a noise component, or as a noise component bysubtracting a low-frequency component 921 which does not include thenoise component and is extracted by an LPF (low-pass filter) processing,from the captured image; and performing a noise removal processing i.e.a noise component removal processing such as a coring processing withrespect to the high-frequency component 922 to remove the noisecomponent. It is highly likely that the high-frequency component 922which is isolated as the frequency component including noise may includea frequency component as a real component of the image i.e. ahigh-frequency component 9222, in other words, frequency noise issuperimposed. The high-frequency component 9222 is a component whosefrequency is higher than that of the low-frequency component 921, but islower than that of a high-frequency component 9221, or a high-frequencycomponent 9223 substantially equal to the high-frequency component 9221in frequency level. If the noise removal is performed by theconventional method, the real image component may also be removed withthe noise component.

FIG. 26 is a diagram showing the high-frequency components 9221, 9222,and 9223 on the same drawing for convenience of explanation. As shown inFIG. 26, the high-frequency components 9221 and 9223 are noisecomponents, and the high-frequency component 9222 is an edge component,which is a real image component. In the case where a noise removalamount is set, and the noise component in the range of the noise removalamount is removed, it is possible to remove the entirety of theright-side-located high-frequency component 9223 without any processing.However, concerning the left-side-located high-frequency components 9221and 9222, it is impossible to remove the noise component at the portionsindicated by e.g. the reference numerals 923, 924, and 925, becausethese portions 923, 924, and 925 are out of the range of the noiseremoval amount. Further, if the portion of the high-frequency component9221 indicated by e.g. the reference numeral 926 is attempted to beremoved, because the portion 926 is in the range of the noise removalamount, an edge portion indicated by the reference numeral 927 i.e. thehigh-frequency component 9222 is also removed.

In view of the above, as a method for removing a noise component whilepreserving an edge component, there is disclosed a technique in e.g.Japanese Unexamined Patent Publication No. 2001-298621 (D1). D1discloses a method comprising: generating a low-frequency componentwhose edge component is preserved by an epsilon filter processing usingε filters arranged in series; and performing a coring processing withrespect to a high-frequency component generated by subtracting the lowfrequency component from an original image for noise removal. Theepsilon filter processing is not a processing to be executed byhierarchical steps, which will be described later. With use of thetechnique, the low-frequency component can be extracted from theoriginal image in such a manner that the edge component is included inthe low-frequency component. In other words, the edge component is notincluded in the high-frequency component to be removed as the noisecomponent. Accordingly, the edge component is preserved without beingaffected by the noise removal processing. However, in this technique, ifsuperimposed noise that the noise component is superimposed over thereal image component is included, it is impossible to exclusively removethe noise component from the superimposed noise.

In light of the above drawback, e.g. Japanese Unexamined PatentPublication No. 2000-134625 (D2) discloses a method comprising: dividinga high-frequency component including a superimposed noise component intoplural frequency components; and isolating the noise component from thereal image component for noise removal. Specifically, this methodcomprises: in performing a frequency band division processing ofdividing an input image into plural frequency components i.e. frequencyband components, referring to a high frequency component i.e. a middlefrequency component generated by a succeeding division processing withrespect to a high-frequency component to be removed as a noisecomponent; and changing the currently generated high-frequency componentbased on a normalization coefficient to be used in edge detection, whichhas been calculated based on a maximal value of the succeedinglygenerated high-frequency component. The normalization coefficient is acoefficient having a property that the coefficient is set to a largevalue if the edge component is detected, and is set to a small value ifthe edge component is not detected. The edge component is preserved bymultiplying the normalization coefficient with the high-frequencycomponent to be removed as the noise component, and the noise componentother than the edge component is removed. Thus, the noise component isisolated from the real image component in the high-frequency componentby the frequency band division processing in calculating thenormalization coefficient. Thus, the noise component is exclusivelyremoved, even if the superimposed noise is included.

In recent years, as the high quality image is demanded in the technicalfield of image sensing apparatuses such as digital cameras, there is atask of increasing a luminance range i.e. a dynamic range of a subjectto be handled by an image sensor. Concerning the technique of increasingthe dynamic range, there are known e.g. an image sensor usinglogarithmic compression i.e. a logarithmic sensor, and alinear-logarithmic sensor. The logarithmic sensor is constructed in sucha manner that an electric signal commensurate with an incident lightamount is logarithmically transformed and outputted. Thelinear-logarithmic sensor has a photoelectric conversion characteristicincluding a linear characteristic that an electric signal is linearlytransformed and outputted in a low luminance area, and a logarithmiccharacteristic that the electric signal is logarithmically transformedand outputted in a high luminance area. An image which is captured bythe linear-logarithmic sensor, and has the linear characteristic and thelogarithmic characteristic of the linear-logarithmic sensor is called a“linear-logarithmic image”. With use of these image sensors, a naturallylogarithmically transformed output is obtained with respect to theincident light amount. Accordingly, these image sensors are advantageousin capturing an image having a wider dynamic range, by a one-timeexposure operation, as compared with an image sensor having aphotoelectric conversion characteristic merely with a linearcharacteristic.

In the current technology, whereas a wide dynamic range is secured in animaging system, as an imaging device such as the linear-logarithmicsensor has been developed, a wide dynamic range is not secured in adisplay system i.e. an image display device such as a monitor, ascompared with the imaging system. Even if a wide dynamic range issecured in the imaging system, the effect of the wide dynamic rangecannot be satisfactorily exhibited on the display system having arelatively narrow dynamic range, as compared with the imaging system. Inother words, a captured image with a wide dynamic range is compressed inconformity with the dynamic range of the display system such as themonitor. Accordingly, a resultantly obtained image has a low contrast,which obstructs proper reproduction i.e. display of the captured image.

In view of the above, it is required to perform a gradation conversionprocessing i.e. a contrast emphasis processing such as dynamic rangecompression processing of e.g. extracting an illumination component anda reflectance component from a captured image, and compressing theillumination component so that the captured image with a wide dynamicrange is displayed in the dynamic range of the display system. In thespecification and the claims, an image with a wide dynamic range whichrequires a gradation conversion processing such as a dynamic rangecompression processing to reproduce the image on a display system havinga narrow dynamic range, and consequently requires a gradation conversionprocessing such as a dynamic range compression processing with a largercompression rate e.g. a larger amplitude rate or a larger gradationconversion rate, as compared with an ordinary compression processing, isreferred to as a “wide dynamic range image”.

In the wide dynamic range image, the noise component as well as the realimage component is greatly amplified by the gradation conversionprocessing. As a result, the noise is emphasized, as compared with animage having an ordinary dynamic range. (not a wide dynamic range) i.e.an ordinary dynamic range image or a standard dynamic range image. Thetechniques disclosed in D1 and D2 involve the following drawbacks, whichare not involved in processing the ordinary dynamic range image by theconventional noise removal processing.

Specifically, in the technique disclosed in D1, it is impossible toisolate the noise component from the superimposed noise in a conditionthat the noise component is superimposed on the real image component,and accordingly, it is impossible to exclusively remove the noisecomponent from the superimposed noise. However, as far as the ordinarydynamic range image is processed, a serious drawback is not involved,because the noise component with respect to the real image component isnegligibly small. However, in the case where the wide dynamic rangeimage is processed, it is highly likely that the noise component withrespect to the real image component may be significantly large. As aresult, the noise component in the image may be intolerably large.

In the technique disclosed in D2, as mentioned above, the succeedinglygenerated high-frequency component in the frequency band divisionprocessing is referred to for calculating the normalization coefficientto be used in edge detection. The high-frequency component is acomponent to be isolated in extracting the noise component in the noiseremoval processing. Therefore, it is highly likely that the noisecomponent may remain in the high-frequency component. The edge componentis preserved by referring to the high-frequency component. Accordingly,the noise component may be erroneously detected and preserved as theedge component. If such a phenomenon occurs, the noise component mayadversely affect the image because it is highly likely that the noisecomponent is non-negligibly large in the wide dynamic range image.

SUMMARY OF THE INVENTION

In view of the above problems residing in the conventional examples, itis an object of the present invention to provide an image processingdevice that enables to finely remove a noise component with less or noinfluence of the noise component in an image (i.e. to secure improvednoise removal performance), and enables to finely preserve an edgecomponent without likelihood that the noise component may be erroneouslydetected and preserved as the edge component (i.e. to secure improvededge preservation performance), and consequently enables to obtain ahigh-quality image, as well as an image processing method, and an imagesensing apparatus using the image processing method.

An image processing device according to an aspect of the inventionincludes: a frequency divider for performing a frequency divisionprocessing of dividing an input image into a plurality of frequencycomponents each having a frequency band; a noise remover for performinga noise component removal processing of removing a noise component froma high-frequency component in the frequency components in the respectivefrequency bands obtained by the frequency division processing by thefrequency divider; an edge preservation information calculator fordetecting an edge intensity based on a low-frequency component in thefrequency components in the respective frequency bands obtained by thefrequency division processing by the frequency divider, and calculatingedge preservation information relating to a degree of preserving an edgecomponent based on the detected edge intensity; an edge preservingsection for preserving the edge component in the high-frequencycomponent, based on the edge preservation information calculated by theedge preservation information calculator; and a frequency synthesizerfor synthesizing the high-frequency component whose noise component isremoved by the noise remover and whose edge component is preserved bythe edge preserving section, and the low frequency component, in each ofthe frequency bands.

These and other objects, features and advantages of the presentinvention will become more apparent upon reading the following detaileddescription along with the accompanying drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram primarily showing an imaging process to beexecuted by a digital camera, as an example of an image sensingapparatus to which an image processing device of a first embodiment ofthe invention is applied.

FIG. 2 is a diagram schematically showing a CMOS image sensor, as atwo-dimensional MOS solid-state image sensor, as an example of an imagesensor shown in FIG. 1.

FIG. 3 is a circuit diagram showing an arrangement of each of pixelsconstituting the image sensor shown in FIG. 2.

FIG. 4 is a graph showing a photoelectric conversion characteristic ofthe image sensor shown in FIG. 1.

FIG. 5 is a functional block diagram showing a circuit configurationprimarily relating to a noise removal processing to be executed by animage processor shown in FIG. 1.

FIG. 6 is a diagram for describing a frequency division processing to beexecuted by a frequency divider shown in FIG. 5, specifically, apartially enlarged view showing the frequency divider shown in FIG. 5.

FIG. 7 is a diagram for describing a high frequency generationprocessing to be executed by a high frequency generator shown in FIG. 5,specifically, a partially enlarged view of the high frequency generatorshown in FIG. 5.

FIG. 8 is a diagram for describing an edge detection processing to beexecuted by an edge detecting unit shown in FIG. 5, and the noiseremoval processing to be executed by a noise removing unit shown in FIG.5, specifically, a partially enlarged view showing the edge detectingunit and the noise removing unit shown in FIG. 5.

FIG. 9 is a graph showing an example of an edge preserving coefficientto be calculated by the edge detecting unit.

FIG. 10 is a graph showing a coring characteristic of a coringprocessing to be executed by the noise removing unit.

FIG. 11 is a diagram for describing a high-frequency image synthesisprocessing and an upsampling processing to be executed by a frequencysynthesizing unit shown in FIG. 5, specifically, a partially enlargedview of the frequency synthesizing unit shown in FIG. 5.

FIG. 12 is a flowchart showing an operation to be executed by the noiseremoval processing in the first embodiment.

FIG. 13 is a functional block diagram showing a circuit arrangementprimarily relating to a noise removal processing to be executed by animage processor of a digital camera in a second embodiment of theinvention.

FIG. 14 is a partially enlarged view of an HPF unit, an edge detectingunit, and a subtracting unit shown in FIG. 13.

FIG. 15 is a partially enlarged view of a noise removing unit and anadding unit shown in FIG. 13.

FIG. 16 is a flowchart showing an operation of the noise removalprocessing to be executed in the second embodiment.

FIGS. 17 and 18 are diagrams for conceptually describing a frequencydivision processing to be executed in the first and the secondembodiments.

FIG. 19 is a functional block diagram showing a circuit configurationprimarily relating to a noise removal processing to be executed by animage processor of a digital camera in a third embodiment of theinvention, specifically, a partially enlarged view of a subband divisionfunctioning part in the image processor.

FIG. 20 is a functional block diagram showing a circuit configurationprimarily relating to the noise removal processing to be executed by theimage processor of the digital camera in the third embodiment,specifically, a partially enlarged view of a subband synthesisfunctioning part in the image processor.

FIG. 21 is a functional block diagram showing a circuit configuration ofan NR section shown in FIG. 20.

FIG. 22 is a schematic diagram for describing as to how an input imageis divided into a high-frequency image and a low-frequency image by aWavelet transformation i.e. a frequency division processing in the thirdembodiment.

FIG. 23 is a flowchart showing an operation of the noise removalprocessing to be executed in the third embodiment.

FIG. 24 is a schematic diagram for describing an edge component of animage in the case where an LPF processing using an epsilon filter isperformed according to the conventional art.

FIGS. 25 and 26 are schematic diagrams for describing a conventionalnoise removal processing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment

FIG. 1 is a block diagram primarily showing an imaging process to beexecuted by a digital camera, as an example of an image sensingapparatus to which an image processing device of a first embodiment ofthe invention is applied. As shown in FIG. 1, the digital camera 1includes a lens section 2, an image sensor 3, an amplifier 4, an A/Dconverter 5, an image processor 6, an image memory 7, a main controller8, a monitor section 9, and an operation section 10.

The lens section 2 functions as a lens aperture for taking subject lighti.e. a light image, and serves as an optical lens system for guiding thesubject light toward the image sensor 3 disposed in the interior of thecamera body. The optical lens system is e.g. a lens group arranged inseries along an optical axis L of the subject light, and including e.g.a zoom lens, a focus lens, and other fixed lens blocks. The lens section2 includes a diaphragm (not shown) and a shutter (not shown) foradjusting a light transmission amount through the lens elements. Thelens section 2 is so constructed that the diaphragm and the shutter aredriven by the main controller 8.

The image sensor 3 is adapted to photoelectrically convert the subjectlight into image signals of respective colors components of R (red), G(green), and B (blue) commensurate with the light amount of the subjectimage formed in the lens section 2 for outputting the image signals tothe amplifier 4 to be described later. In this embodiment, the imagesensor 3 is a solid-state image sensor having a photoelectric conversioncharacteristic having different characteristic areas i.e. aphotoelectric conversion characteristic, as shown in FIG. 4, including:a linear characteristic area where an output pixel signal i.e. an outputelectrical signal generated by photoelectric conversion is linearlytransformed and outputted when the incident luminance of the light imageobtained by the image sensor 3 is low, namely, the light image is dark;and a logarithmic characteristic area where the output pixel signal islogarithmically transformed and outputted when the incident luminance ofthe light image is high, namely, the light image is bright. In otherwords, the image sensor 3 has a photoelectric conversion characteristicwith a linear curve in a low luminance range, and a logarithmic curve ina high luminance range. The image sensor 3 is also called as a“linear-logarithmic sensor”. The image sensor 3 is also selectivelysettable to a photoelectric conversion characteristic merely having alinear characteristic or a logarithmic characteristic, in place of thephotoelectric conversion characteristic having the linear/logarithmiccharacteristic. Also, a switching point (hereinafter, called as“inflection point”) between the linear characteristic area and thelogarithmic characteristic area of the photoelectric conversioncharacteristic is controllable based on a predetermined control signalapplied to the respective pixel circuits of the image sensor 3 by e.g.changing a magnitude of a voltage difference between high (Hi) and low(Low) of a certain DC voltage to be used in driving a MOSFET to bedescribed later.

FIG. 2 is a diagram schematically showing a CMOS image sensor, as atwo-dimensional MOS solid-state image sensor, as an example of the imagesensor 3. In FIG. 2, G11 through Gmn denote pixels arranged in a matrixpattern. A vertical scanning circuit 301 and a horizontal scanningcircuit 302 are arranged near a perimeter of the pixel sectionconstituted of the pixels G11 through Gmn. The vertical scanning circuit301 sequentially scans signal lines 304-1, 304-2, . . . , and 304-n(generically called as “row signal lines 304”) arranged in a rowdirection. The horizontal scanning circuit 302 sequentially readsphotoelectric conversion signals outputted from the respective pixelsthrough output signal lines 306-1, 306-2, . . . , and 306-m (genericallycalled as “output signal lines 306”) in a horizontal direction, pixel bypixel. An electric power is supplied to the pixels through a powersource line 305 from a power source (not shown). Signal lines e.g. clocksignal lines other than the row signal lines 304 and the output signallines 306 are connected to the pixels, but are not illustrated in FIG.2.

Constant current sources 307-1, 307-2, . . . , and 307-m (genericallycalled as “constant current sources 307”) each constituting anamplifying circuit with a transistors T12 in pair are provided withrespect to the output signal lines 306-1, 306-2, . . . , and 306-m,respectively. A resistor or a transistor may constitute the amplifyingcircuit, in place of the constant current source 307. Image data to beused in capturing an image, and correction data to be used in resetting,which are outputted from the pixels via the output signal lines 306, aresequentially outputted to selecting circuits i.e. sample-and-holdcircuits 308-1, 308-2, . . . , and 308-m (generically called as“selecting circuits 308”). The image data and the correction data areoutputted to the selecting circuits 308 row by row forsampling-and-holding. The sampled-and-held image data and correctiondata are outputted to a correcting circuit 309 column by column. Then,the image data is corrected by the correcting circuit 309, based on thecorrection data to remove sensitivity variation. After the sensitivityvariation with respect to each of the pixels is corrected by thecorrecting circuit 309, the image data is serially outputted pixel bypixel.

FIG. 3 is a circuit diagram showing an arrangement of each of the pixelsG11 through Gmn shown in FIG. 2. As shown in FIG. 3, each of the pixelsof the image sensor 3 includes a photodiode PD1, transistors T10 throughT13 as MOSFETs (Metal Oxide Semiconductor Field Effect Transistors), andan FD (Floating Diffusion). The transistors T10 through T13 each is anN-channel MOSFET in this embodiment. VDD, φRSB, φRST, φTX, and φV denotesignals i.e. voltages to be applied to the transistors, and GND denotesthe ground.

The photodiode PD1 is a light sensing element i.e. a photoelectricconverter, and is adapted to generate an electric signal i.e. aphotocurrent IPD1 commensurate with the amount of incident light from asubject. The transistor T12 and the constant current source 307 in pairconstitute an amplifying circuit i.e. a source follower amplifier foramplifying a source follower. The transistor T12 amplifies a currentwith respect to a voltage V1OUT, which will be described later. Thetransistor T13 is a transistor i.e. a row selection transistor for asignal readout operation, and is operated as a switch to be turned onand off in accordance with the voltage i.e. the signal φV to be appliedto the gate. The source of the transistor T13 is connected to the outputsignal line 306. When the transistor T13 is turned on, a currentamplified by the transistor T12 is outputted to the output signal line306 as an output current.

The transistor T10 is operated as a switch to be turned on and off inaccordance with the voltage to be applied to the gate of the transistorT10. The transistor T10 functions as a transfer gate for switching overbetween transfer and non-transfer of the photocurrent IPD1 i.e. anelectric charge generated in the photodiode PD1 to the FD in accordancewith an on/off operation corresponding to high/low of the gatepotential. The photocurrent IPD1 generated in the photodiode PD1 flowsto a parasitic capacitance of the photodiode PD1 to accumulate theelectric charge, and a voltage is generated in accordance with theamount of the accumulated charge. In this condition, if the transistorT10 is in an on-state, the charge i.e. a negative charge accumulated inthe parasitic capacitance is moved to the FD. The FD is a chargeretainer for temporarily retaining the charge i.e. the signal charge.The FD serves as a capacitor for converting the retained charge into avoltage, i.e. performing a charge/voltage conversion.

The transistor T11 i.e. a reset gate transistor is adapted to switchover between application and non-application of a reset bias to the FDin accordance with an on/off operation corresponding to high/low of thegate voltage of the transistor T11. For instance, if the transistor T11is in an on-state, the transistor T10 is also in an on-state, and areset bias is applied between φRSB and GND, with the transistor T11, FD,the transistor T10, and the photodiode PD1 being interposedtherebetween. By setting the gate voltage to a Mid potential i.e. apotential at an intermediate level, a linear transformation and alogarithmic transformation are performed by charge/voltage conversion bythe FD and the transistor T11, respectively, concerning the charge (i.e.the current flowing in the FD) moving from the photodiode PD1 to the FD.

In the above condition, a current i.e. a reset current depending on theMid potential flows through the transistor T11, and the source of thetransistor T11 is set to a potential depending on the reset current. Ifthe potential by the charge moving from the photodiode PD1 is smallerthan the source potential of the transistor T11 depending on the Midpotential, in other words, if the luminance of the subject whose imageis to be captured is low, i.e. the subject is dark, and the amount oflight incident onto the photodiode PD1 is small, a charge/voltageconversion as a linear transformation is performed by the FD. If, on theother hand, the potential by the charge moving from the photodiode PD1exceeds the source potential of the transistor T11, in other words, ifthe luminance of the subject whose image is to be captured is high, i.e.the subject is bright, and the amount of light incident onto thephotodiode PD1 is large, a charge/voltage conversion as a logarithmictransformation is performed by the transistor T11.

With this arrangement, a voltage as a linear output by an integrationvalue of the photocurrent IPD1 in the FD, or a voltage as a logarithmicoutput by current/voltage conversion depending on the photocurrent IPD1in the transistor T11 is obtained at a connection node i.e. as theoutput V1OUT between the FD and the transistor T12. Specifically, theoutput value in the linear characteristic area of the photoelectricconversion characteristic is the integration value of the photocurrentIPD1 in the FD. However, concerning the logarithmic characteristic area,a current substantially equal to the photocurrent IPD1 flows in thetransistor T11 in an area where the potential by the charge accumulatedin the FD exceeds the source current of the transistor T11 i.e. thereset gate, and the voltage value obtained by current/voltage conversionof the photocurrent IPD1 in the transistor T11 is obtained in the FD asan output value. In other words, the charge whose signal islogarithmically compressed is accumulated in the parasitic capacitance.The current/voltage conversion in the transistor T11 corresponds to theaforementioned logarithmic transformation. In this state, when thetransistor T13 is turned on, an amplification current in the transistorT12 depending on the respective voltages is derived as an output currentthrough the output signal line 306 via the transistor T13. Thus, anoutput signal subjected to linear transformation or logarithmictransformation in accordance with a subject luminance i.e. an incidentluminance onto the image sensor 3 i.e. a wide dynamic range image isobtained by the image sensor 3.

In this embodiment, the N-channel MOSFET is provided in each of thepixels of the image sensor 3. Alternatively, a P-channel MOSFET may beprovided in each of the pixels. Alternatively, the image sensor 3 may bea CMOS image sensor for performing a linear transformation and alogarithmic transformation by utilizing a sub-threshold characteristicof P-channel or N-channel MOSFET, in place of the CMOS image sensor forperforming a linear transformation and a logarithmic transformationusing the FD. The image sensor 3 maybe a VMIS image sensor, a CCD imagesensor, or a like sensing device, in place of the CMOS image sensor.

As mentioned above, the image sensor 3 is capable of performing a widedynamic range imaging such that an image with a wider dynamic range isobtained, i.e. capturing an image having subject luminance informationin a broad luminance range from a low luminance to a high luminance. Theimage sensor has such a photoelectric conversion characteristic that anoutput value of the image sensor with respect to the subject luminancein the broad luminance range from the low luminance to the highluminance does not exceed a maximal output value of the image sensor.The image sensor 3 i.e. a linear-logarithmic sensor has a photoelectricconversion characteristic having a feature that the photoelectricconversion characteristic in the high luminance area is logarithmicallychanged in a gradually increasing manner so that the sensor output doesnot exceed the maximal output value in a wider luminance range i.e. asubject luminance in a wider luminance range can be captured within theoutput range of the image sensor. In other words, the image sensor 3 hasa photoelectric conversion characteristic that an increase in sensoroutput level relative to an increase in luminance level is moderate, asthe subject luminance approaches the high luminance area where thesensor output value is sharply increased. Alternatively, various imagesensors other than the image sensor 3 provided with the aforementionedphotoelectric conversion characteristic may be applied, as far as suchan image sensor is capable of performing a wide dynamic range imaging.

The amplifier 4 is adapted to amplify the image signal outputted fromthe image sensor 3. The amplifier 4 includes e.g. an AGC (auto gaincontrol) circuit to adjust a gain of the image signal outputted from theimage sensor 3. The amplifier 4 may include a CDS (correlation doublesampling) circuit for reducing sampling noise of the image signal as ananalog value, in addition to the AGC circuit. The AGC circuit also has afunction of compensating insufficiency of output level concerning animage to be photographed in photographing e.g. a subject with anexceedingly low luminance, in the case where a proper exposure is notobtained. The gain with respect to the AGC circuit is set by the maincontroller 8. The A/D converter 5 performs A/D conversion of convertingan image signal of an analog value amplified by the amplifier 4 i.e. ananalog signal into an image signal of a digital value i.e. a digitalsignal. The A/D converter 5 converts a pixel signal obtained byreceiving light on the respective pixels of the image sensor 3 intopixel data of e.g. 12 bits.

The image processor 6 performs various image processing with respect tothe image signal obtained by the A/D conversion, specifically, colorprocessing such as color interpolation, color correction, and colorspace conversion, or white balance correction, noise removal processing,or dynamic range compression processing. A feature of this embodimentparticularly resides in the noise removal processing among the variousimage processing. The noise removal processing will be described laterin detail.

The image memory 7 is a memory including an ROM (Read Only Memory) andan RAM (Random Access Memory). The image memory 7 is adapted to storedata including raw image data before being processed by the imageprocessor 6, and image data during processing or after being processedby the image processor 6 or the main controller 8.

The main controller 8 includes an ROM for storing various controlprograms, an RAM for temporarily storing various data, and a centralprocessing unit (CPU) for reading the control program and the like fromthe ROM for execution. The main controller 8 controls entire operationsof the digital camera 1. The main controller 8 calculates controlparameters necessary for operating the respective parts of the digitalcamera 1, e.g., an exposure amount control parameter to be used insetting an optimal exposure amount in photographing, or a dynamic rangecontrol parameter to be used in setting an optimal dynamic range, basedon various signals from the parts of the digital camera 1 such as theimage sensor 3 and the operation section 10; and controls the operationsof the respective parts by transmitting the control parameters thereto.The exposure amount control is executed by e.g. adjustment of theopening amount of the diaphragm, or adjustment of the shutter speed ofthe mechanical shutter, or control of an integration time i.e. anexposure time for charge accumulation, which is controlled by a resetoperation with respect to the image sensor 3. The dynamic range controlis executed by e.g. controlling the switching point i.e. the inflectionpoint between the linear characteristic area and the logarithmiccharacteristic area of the image sensor 3.

The main controller 8 controls the image sensor 3 and the lens section 2including the diaphragm and the shutter to perform an imaging operationvia e.g. a timing generator or a driving section (both of which are notshown), based on the aforementioned various control parameters. The maincontroller 8 also controls the monitor section 9 to display e.g. imagedata stored in the image memory 7, and also controls the image processor6 to perform the below-mentioned image processing control concerning thenoise removal processing, in addition to the aforementioned controls.

The monitor section 9 includes a liquid crystal display (LCD), e.g. acolor liquid crystal display device, provided on a rear surface of thedigital camera 1. The monitor section 9 is adapted to display an imagecaptured by the image sensor 3, i.e., an image processed by the imageprocessor 6, an image stored in the image memory 7, and the like. Theoperation section 10 is adapted to allow the user to designate i.e.input an operation with respect to the digital camera 1. The operationsection 10 is provided with various operation switches i.e. operationbuttons e.g. a power source switch, a release switch, a mode settingswitch for setting various photographing modes, and a menu selectionswitch. For instance, when the release switch is pressed down i.e.turned on, a series of photographing operations comprising: sensingsubject light by the image sensor 3; applying a predetermined imageprocessing to image data acquired by the sensing operation; andrecording the processed image data in the image memory 7 or a likedevice are executed.

As mentioned above, a wide dynamic range image obtained by the sensingoperation of the image sensor 3 is inputted to the image processor 6.Hereinafter, the wide dynamic range image inputted to the imageprocessor 6 is also called as an “input image”, according to needs. Inthe following, the noise removal processing to be executed with respectto the inputted wide dynamic range image by the image processor 6 isdescribed. FIG. 5 is a functional block diagram showing a circuitconfiguration primarily relating to the noise removal processing to beexecuted by the image processor 6. As shown in FIG. 5, the imageprocessor 6 includes a frequency divider 61, a high frequency generator64, an edge detecting unit 65, a noise removing unit 66, and a frequencysynthesizing unit 69. Hereinafter, the high frequency generator 64 isalso called as “HPF unit 64” according to needs.

First, the aforementioned functional parts are briefly described. Thefrequency divider 61 divides an input image 610 inputted to the imageprocessor 6 i.e. a two-dimensional wide dynamic range image into pluralkinds of frequency components i.e. frequency bands or frequency bandcomponents by performing an LPF processing and a downsampling processingwith respect to the input image 610 in hierarchical processing steps. Bythe process of isolating the frequency components i.e. a frequencydivision processing or a band division processing, an image with alow-frequency component i.e. a low-frequency image is generated eachtime the processing step is implemented. The high frequency generator 64generates a high-frequency image in each of the processing steps bysubtracting the LPF-processed low-frequency image from the image beforethe LPF processing is performed in each of the processing steps in thefrequency division processing to be executed by the frequency divider61.

The edge detecting unit 65 performs a detection processing concerning anedge component i.e. an edge detection processing with respect to theLPF-processed low-frequency image in each of the processing steps. Thenoise removing unit 66 performs a noise removal processing of removing anoise component from an image including a high-frequency component i.e.a high-frequency image, which has been generated by the high frequencygenerator 64. The frequency synthesizing unit 69 generates a widedynamic range image free of noise, as a final output image to beoutputted from the image processor 6, by performing a synthesisprocessing i.e. an adding processing and an upsampling processing withrespect to a high-frequency image obtained after the noise removalprocessing is performed in each of the processing steps, and thelow-frequency image obtained in the final processing step of theprocessing steps, which has been generated by the frequency divisionprocessing by the frequency divider 61.

The constructions and operations of the aforementioned parts will bedescribed in detail, referring to FIGS. 6 through 11. FIG. 6 is adiagram for describing the frequency division processing to be executedby the frequency divider 61, and specifically, is a partially enlargedview of the frequency divider 61 shown in FIG. 5. As shown in FIG. 6,the frequency divider 61 has an LPF unit 62 for performing the LPFprocessing, and a DS unit 63 for performing the downsampling (DS)processing. The LPF unit 62 includes LPF sections to be used in therespective processing steps i.e. an LPF section 621 to be used in aprocessing step 1, an LPF section 622 to be used in a processing step 2,. . . , an LPF section 623 to be used in a processing step (n-1); and anLPF section 624 to be used in a processing step n. The LPF sections 621through 624 each performs an LPF processing using a low-pass filter withrespect to the input image in each of the processing steps, and extractsa low-frequency image in each of the processing steps. For instance, inthe processing step (n-1), the LPF section 623 performs an LPFprocessing with respect to an input image 6231, and extracts alow-frequency image 6232. The input image in this section means an imageto be inputted in the corresponding processing step. The input image inthe processing step 1 is image data representing an image which has beensent from the image sensor 3 and has not been subjected to an LPFprocessing. However, the input image in the processing step 2 andthereafter is an LPF-processed low-frequency image. The low-frequencyimage extracted by the LPF processing is an image to be used in edgedetection i.e. an image to be used in calculating an edge preservationcoefficient, which will be described later.

The DS unit 63 includes DS sections to be used in the respectiveprocessing steps i.e. a DS section 631 to be used in the processing step1, a DS section 632 to be used in the processing step 2, . . . , and aDS section 633 to be used in the processing step (n-1). Since theprocessing step n is a final processing step, the DS unit 63 does notincludes a DS section corresponding to the processing step n. The DSsections 631 through 633 each performs a DS processing i.e. one-halfdownsampling of reducing the image size i.e. the pixel number toone-half in row and column directions by e.g. pixel interpolation, withrespect to the low-frequency image extracted by the LPF processing ineach of the processing steps; and outputs the downsampled image to theLPF section in the succeeding processing step. For instance, in theprocessing step (n-1), the DS section 633 performs a DS processing withrespect to the low-frequency image 6232 after the LPF processing isperformed by the LPF section 623, and generates an input image 6241 tobe processed in the processing step n. In this embodiment, one-halfdownsampling is performed. Alternatively, the DS processing may beexecuted with an arbitrary magnification ratio e.g. one-third,one-fourth, and the like.

Any value is applied to the processing step number i.e. the value “n” inthe processing step n. In this embodiment, the number “n” is set to e.g.4 (n=4). Hereinafter, description is made on the premise that theprocessing step n is the processing step 4 according to needs. Also, inthis embodiment, in the case where the processing step n=4, the LPFprocessing to be executed by the respective LPF sections is a processingusing an LPF of a filter size whose tap number is e.g. “7”, in otherwords, a 7×7 LPF processing. The relation between the processing stepnumber and the tap number may be set in such a manner that in performing5×5 LPF processing by the LPF processing using the tap number smallerthan “7” e.g. “5”, the processing step number n may be set to a valuelarger than “4” e.g. “8”. This is because, as the tap number isdecreased, the degree of image smoothing to be obtained each time theLPF processing is performed is decreased, which requires increasing thenumber of times of performing the LPF processing i.e. the number ofprocessing steps. Alternatively, it may be needless to set theprocessing step number and the tap number to satisfy the aforementionedrelation. In other words, the number of the hierarchical processingsteps and the tap number of the LPF may be set in such a manner that alow-frequency image to be obtained by the LPF processing by the LPFsection is free of residual noise, in other words, a low-frequencycomponent free of noise component can be extracted in the processingstep n.

In this way, the frequency divider 61 performs the multiple LPFprocessing and the multiple downsampling processing in the multipleprocessing steps from the uppermost processing step 1 to the lowermostprocessing step 4 stepwise, whereby the input image 610 i.e. the widedynamic range image is divided into plural frequency images. Repeatedlyperforming the LPF processing and the DS processing is equivalent toperforming the LPF processing while increasing the tap number stepwise.

FIG. 7 is a diagram for describing a high frequency generationprocessing to be executed by the high frequency generator 64, and is apartially enlarged view of the high frequency generator 64 shown in FIG.5. As shown in FIG. 7, the high frequency generator 64 includes HPFsections 641, 642, 643, and 644 to be used in the processing steps 1through n i.e. 1 through 4, respectively. The HPF sections 641 through644 each generates i.e. extracts a high-pass-filter (HPF) processedhigh-frequency image by subtracting a low-frequency image from the inputimage sent from the frequency divider 61. In other words, the HPF unit64 obtains a high-frequency image by subtracting an image i.e. alow-frequency image which has been inputted to the LPF unit 62 and issubjected to an LPF processing, from the image before being inputted tothe LPF section 62 i.e. before being subjected to the LPF processing.The processing for generating the high-frequency image by the highfrequency generator 64 is called a high frequency generation processing.

For instance, in the processing step (n-1) i.e. in the processing step3, after the input image 6231, which has been outputted from the DSsection 632 in the processing step (n-2) preceding the processing step(n-1) i.e. in the processing step 2, and the low-frequency image 6232,which has been outputted from the LPF section 623 are inputted to theHPF section 643, the HPF section 643 subtracts the low-frequency image6232 from the input image 6231. As a result of the subtraction, the HPFsection 643 extracts and outputs a high-frequency image 6432.High-frequency images generated by the respective HPF sections 641through 644 are an image obtained by isolating a high-frequencycomponent including a noise component i.e. a superimposed noisecomponent from the input image in each of the processing steps.Accordingly, a noise removal processing is performed with respect toeach of the high-frequency images in the succeeding processing step. Itis possible to isolate the superimposed noise component that could notbe isolated by a one-time HPF processing, by performing the multiple HPFprocessing in the multiple processing steps.

FIG. 8 is a diagram for describing the edge detection processing to beexecuted by the edge detecting unit 65, and the noise removal processingto be executed by the noise removing unit 66, and is a partiallyenlarged view of the edge detecting unit 65 and the noise removing unit66 shown in FIG. 5. As shown in FIG. 8, the edge detecting unit 65includes edge detectors 651, 652, 653, and 654 to be used in theprocessing steps 1 through 4, respectively, and the noise removing unit66 includes noise removers 661, 662, 663, and 664 to be used in theprocessing steps 1 through 4, respectively. The edge detecting unit 65and the noise removing unit 66 perform a noise removal processing whilepreserving an edge component in the high-frequency image, which is sentfrom the HPF unit 64 and includes the noise component. Examples of theedge detection processing to be executed by the edge detecting unit 65and the noise removal processing to be executed by the noise removingunit 66 are described in the following.

<Edge Detection Processing>

As an edge detection processing, the edge detectors 651 through 654 eachcalculates an edge intensity of the low-frequency image by performing afilter processing with respect to the low-frequency image, using an edgeintensity detection filter e.g. a Sobel filter i.e. an edge intensitydetection filter processing in each of the processing steps; andcalculates an edge preservation coefficient E based on the informationrelating to the calculated edge intensity. The information relating tothe calculated edge preservation coefficient E is outputted to the noiseremovers 661 through 664. For instance, the edge detector 653 in theprocessing step 3 obtains an edge intensity by performing an edgeintensity detection filter processing with respect to the low-frequencyimage 6232 sent from the HPF unit 64, calculates an edge preservationcoefficient E based on the edge intensity, and outputs i.e. sends, tothe noise remover 663, the information relating to the calculated edgepreservation coefficient E, as edge preservation coefficient information6532. The Sobel filter is adapted to detect a gradient between thepixels in the image, based on differentiation to detect an edgeintensity in the image.

(Calculation of Edge Preservation Coefficient)

The edge intensity and the edge preservation coefficient E arecalculated as follows. Specifically, the edge detectors 651 through 654each calculates an edge intensity “e” by performing a filter processingwith respect to each of the low-frequency images, using a 3×3 Sobelfilter, i.e. by performing a 3×3 Sobel filter processing, wherein thetap number is “3”; and calculates the edge preservation coefficient Ewith respect to the edge intensity “e” by the following formula (1).

when e<e1, E=0,

when e1<e<e2, E=(e−e1)/(e2−e1), and

when e>e2, E=1.0   (1)

where e1, e2 is a predetermined threshold value to be used incalculating an edge preservation coefficient, i.e. an edge preservationthreshold value, and e1<e2.

The relation of the formula (1) is expressed in FIG. 9. Referring toFIG. 9, wherein the axis of ordinate represents the edge preservationcoefficient E and the axis of abscissas represents the edge intensity“e”, the edge preservation coefficient E=0, which represents that thedegree of edge preservation is minimal, in the area where the edgeintensity “e” indicated by the reference numeral 655 is smaller than e1i.e. e<e1; the edge preservation coefficient E=1.0, which is a maximalvalue indicating that the degree of edge preservation is maximal, in thearea where the edge intensity “e” indicated by the reference numeral 657is larger than e2 i.e. e>e2; and the edge preservation coefficient E islinearly changed from the value “0” to “1.0”, with a gradient of1/(e2−e1), in the area where the edge intensity “e” indicated by thereference numeral 656 is larger than e1 and smaller than e2 i.e.e1<e<e2.

The edge preservation threshold value e1, e2 is a value relating totexture of the image to be used in determining to what degree the edgeof the image is to be preserved, and is set in advance in each of theprocessing steps so that the edge of the image isoptimally preserved.The edge preservation threshold value e1, e2 may be set in advance ineach of the processing steps, or may be set to a common value throughoutthe processing steps. The latter modification means that the same valueis set, although the value is set in each of the processing steps. Theinformation relating to the edge preservation threshold value e1, e2 isstored in the edge detecting unit 65 i.e. the edge detectors 651 through654. In this embodiment, a 3×3 Sobel filter is used as the edgeintensity detection filter, as mentioned above. Alternatively, a Sobelfilter whose tap number is other than “3” e.g. larger than “3” may beused, or a filter such as a Prewitt filter may be used.

<Noise Removal Processing>

The noise removers 661 through 664 each performs, as a noise removalprocessing, a coring processing with respect to the high-frequency imagein each of the processing steps. In the coring processing, the noiseremovers 661 through 664 each performs an edge component preservationprocessing of preserving an edge component whose coring intensity in thecoring processing is decreased, with respect to a pixel whose edgeintensity “e” is large, by using the edge preservation coefficientinformation calculated in the edge detection processing. Concerning apixel whose edge intensity “e” is small, the degree of edge preservationis decreased by increasing the coring intensity.

<Coring Processing>

In the coring processing, the noise removers 661 through 664 eachremoves a noise component by performing a high-frequency imageconversion processing i.e. an image conversion processing with respectto the high-frequency image inputted to each of the noise removers 661through 664, according to e.g. a characteristic indicated by thereference numeral 665 shown in FIG. 10 i.e. the coring characteristic665. In FIG. 10, the symbol “th” denotes a coring coefficient to be usedin removing a noise component from an image signal. The coringprocessing according to the coring characteristic 665 is a conversionprocessing, wherein the output is 0 when the input is larger than “−th”and smaller than “th” in other words, the data in the area where theinput is larger than “−th” and smaller than “th” is removed as the noisecomponent; the output is a sum of the input and the value “th” when theinput is equal to or smaller than “−th”; and the output is a differencebetween the input and the value “th” when the input is equal to orlarger than “th”.

The aforementioned high-frequency image includes a detail component aswell as the noise component. Accordingly, if an image component in anarea where the absolute value of a targeted pixel is smaller than thecoring coefficient i.e. larger than “−th” and smaller than “th” isremoved by the coring processing without an exception, the detailcomponent may also be removed. In view of this, it may be preferable tointroduce a coring characteristic 666 having a relation: y=kx showingthat the gradient k of the graph satisfies: 0≦k≦1 in the area where theabsolute value of the targeted pixel is larger than “−th” and smallerthan “th”, in place of using the coring characteristic 665; and adjustthe “kn” value so that a processing result with no or less removal of adetail component is obtainable, with a negligibly small noise beingallowed to remain in the image. In the case where the coringcharacteristic 666 is used, a detail component remains unremoved, as thevalue of the input X is approximated to the value of the coringcoefficient “th” (“−th”), as shown by e.g. the reference numeral 6661,6662 in the graphical expression: y=kx. The detail component is an imagein the high-frequency image other than the noise component, and is animage i.e. a frequency component originated from the high-frequencyimage, which represents texture of the image, in contrast to alow-frequency image representing a blur image.

Fine and accurate noise removal can be performed by setting the coringcoefficient “th” (“−th”) to a value depending on the amount of noise inthe high-frequency image in each of the processing steps. Actually, theimage sensor 3 involves noise such as a dark current. In light of thefacts that the noise amount in the high-frequency image is determineddepending on the noise inherent to the sensor, and that the noise amountdiffers among the high-frequency images in the processing steps i.e.depending on the frequency bands, the value of the coring coefficient“th” (“−th”) is set to an optimal value corresponding to an intendednoise removal amount with respect to each of the image sensors 3 to beused in the digital cameras 1, or with respect to each of the processingsteps. The coring coefficient “th” is stored in each of the noiseremovers, as a predetermined fixed value. Alternatively, the coringcoefficient “th” may not be set as the fixed value. For instance, it ispossible to provide a predetermined noise detector for detecting theamount of noise in the high-frequency image in each of the processingsteps so that the coring coefficient “th” is calculated and set inaccordance with the detected noise amount, each time the noise amount isdetected.

(Edge Preservation Processing)

In the edge preservation processing, the noise removers 661 through 664each generates a high-frequency image (called as an “edge preservedimage”), wherein an edge is preserved with an edge preservation amountcorresponding to the edge intensity “e”, in other words, an edgecomponent preservation processing of changing a coring degree i.e. acoring intensity or a coring level depending on the edge intensity “e”is performed, by performing the following operation. Specifically, thenoise removers 661 through 664 each performs a weighting processing i.e.a weighted average processing with respect to a high-frequency imageinputted to each of the noise removers 661 through 664, and an image(hereinafter, called as a “coring image”) obtained as a result of thecoring processing, by the following formula (2), using the edgepreservation coefficient information (see FIG. 9) obtained by the(Calculation of Edge Preservation Coefficient) in the <Edge DetectionProcessing>. By performing the edge component preservation processing,as shown in FIG. 9, the value of a pixel having a larger edgepreservation coefficient E i.e. closer to 1.0, is approximated to thevalue of the pixel of the high-frequency image before the coringprocessing is executed. Thereby, a larger amount of edge component ispreserved. On the other hand, the value of a pixel having a smaller edgepreservation coefficient E i.e. closer to 0, is approximated to thevalue of the pixel of the coring image.

Img_output=(1.0−E)*img_core+E*Img_input   (2)

where Img_output represents an edge preserved image,

Img_core represents a coring image,

Img_input represents a high-frequency image to be inputted to therespective noise removers, and the symbol “*” denotes multiplication(the same definition is also applied to the following description).

Thus, the noise removers 661 through 664 e.g. the noise remover 663performs the coring processing and the edge component preservationprocessing with respect to the high-frequency image 6432 outputted fromthe HPF section 643, based on the edge preservation coefficientinformation 6532 outputted from the edge detector 653, and outputs ahigh-frequency image 6632 with its edge component being preserved, andits noise component being removed.

FIG. 11 is a diagram for describing a synthesis processing and anupsampling processing with respect to a high-frequency image to beexecuted by the frequency synthesizing unit 69, and is a partiallyenlarged view of the frequency synthesizing unit 69 shown in FIG. 5. Asshown in FIG. 11, the frequency synthesizing unit 69 includes asynthesizing unit 67 for performing the synthesis processing, and a USunit 68 for performing the upsampling (US) processing. The synthesizingunit 67 includes synthesizers i.e. adders to be used in the respectiveprocessing steps i.e. a synthesizer 671 to be used in the processingstep 1, a synthesizer 672 to be used in the processing step 2, asynthesizer 673 to be used in the processing step 3 i.e. the processingstep (n-1), and a synthesizer 674 to be used in the processing step 4i.e. the processing step n.

The synthesizers 671 through 674 each generates a frequency-synthesizedimage in each of the processing steps by synthesizing i.e. summing anupsampled image i.e. a frequency-synthesized image to be describedlater, which is obtained in a preceding processing step i.e. in alower-located processing step in FIG. 11, and the noise-removedhigh-frequency image outputted from the noise removers 661 through 664.For instance, in the processing step 3, the high-frequency image 6632outputted from the noise remover 663, and a frequency-synthesized image6731 outputted from a US section 683 are synthesized to generate afrequency-synthesized image 6732. Then, the frequency-synthesized image6732 is inputted to the US section 682 in the upper-located processingstep 2. In the processing step 4 i.e. in the lowermost processing step,not a frequency-synthesized image obtained by synthesizing with an imageoutputted from the lower-located processing step, but a low-frequencyimage 6242 (see FIG. 6) outputted from the LPF section 624 is used. Inother words, the synthesizer 674 synthesizes the low-frequency image6242 and the high-frequency image outputted from the noise remover 664.

The image synthesis processing to be executed by the synthesizers 671through 674 in the respective processing steps is a processing ofsynthesizing a high-frequency image outputted from the noise removingunit 66 with a low-frequency image outputted from the lower-locatedprocessing step (in case of the processing step 4, the low-frequencyimage is outputted from the same processing step); and outputting thesynthesized image to the upper-located processing step, as alow-frequency image. Observing the above processing from thesynthesizers in the processing steps e.g. from the synthesizer 673, thefrequency-synthesized image 6731 generated in the lower-locatedprocessing step is a low-frequency image with respect to thehigh-frequency image 6632.

The US unit 68 includes the US sections to be used in the respectiveprocessing steps i.e. a US section 681 to be used in the processing step1, a US section 682 to be used in the processing step 2, and the USsection 683 to be used in the processing step 3. The US sections 681through 683 each performs 2US processing i.e. double upsampling ofincreasing the image size i.e. the pixel number in row and columndirections by double by e.g. pixel interpolation such as linearinterpolation, with respect to the frequency-synthesized image generatedby the synthesizer in the preceding processing step; and outputs theupsampled image to the synthesizer in the succeeding processing step. Inthe embodiment, the magnification ratio to be used in the US processingis double in correspondence to one-half, which is the magnificationratio used in the DS processing by the DS unit to restore i.e. enlargethe image size which has been reduced to one-half, to the original size.Alternatively, the magnification ratio to be used in the US processingmay be changed depending on the magnification ratio to be used in the DSprocessing. For instance, if the magnification ratio to be used in theDS processing is one-third, the magnification ratio to be used in the USprocessing is triple. Further alternatively, the magnification ratio tobe used in the US processing may be arbitrarily set, independently ofthe magnification ratio to be used in the DS processing.

In the above arrangement, the frequency synthesizing unit 69 performsthe multiple frequency synthesis processing and the multiple USprocessing in the multiple processing steps from the lowermostprocessing step 4 to the uppermost processing step 1 stepwise, andoutputs an output image 690 i.e. a wide dynamic range image from thesynthesizer 671, as an image corresponding to the input image 610inputted to the image processor 6.

FIG. 12 is a flowchart showing an operation of the noise removalprocessing to be executed in the first embodiment. First, apredetermined wide dynamic range image i.e. a linear-logarithmic imageis obtained by a sensing operation or the like of the image sensor 3,and then, the wide dynamic range image i.e. the input image 610 isinputted to the frequency divider 61 of the image processor 6 (Step S1).Then, the frequency divider 61 i.e. the LPF unit 62 and the DS unit 63perform the multiple LPF processing and the multiple DS processing,respectively, with respect to the input image 610 in the multipleprocessing steps from the uppermost processing step to the lowermostprocessing step stepwise to divide the input image 610 into pluralfrequency components i.e. to perform frequency isolation, whereby alow-frequency image is generated in each of the processing steps (StepS2). Then, the HPF unit 64 subtracts an LPF-processed low-frequencyimage from the input image with respect to each of the processing steps,as the low-frequency image is generated in each of the processing steps,whereby a high-frequency image is generated (Step S3).

Then, in each of the processing steps, the edge detecting unit 65 andthe noise removing unit 66 perform the edge detection processing and thenoise removal processing, respectively. Specifically, the edge detectingunit 65 performs an edge intensity detection filter processing withrespect to the low-frequency image inputted to the edge detecting unit65, using an edge intensity detection filter to calculate an edgeintensity of the low-frequency image, and calculates an edgepreservation coefficient E based on the calculated edge intensityinformation. Also, the edge removing unit 66 performs a coringprocessing with respect to the high-frequency image inputted to thenoise removing unit 66 to generate a noise-component-removedhigh-frequency image i.e. a coring image, and generates an edgepreserved image i.e. a high-frequency image whose noise component isremoved and whose edge component is preserved depending on the edgeintensity “e”, by performing a weighting processing with respect to thecoring image and the high-frequency image, using the edge preservationcoefficient information obtained by the edge detecting unit 65 (StepS4). Then, the frequency synthesizing unit 69 repeatedly performs thefrequency synthesis processing and the US processing, from the lowermostprocessing step to the uppermost processing step stepwise, by summingthe low-frequency image obtained in the lowermost processing step, andthe high-frequency images obtained in the respective processing steps.As a result of the summation, a synthesized image whose noise componentis removed and whose edge component is preserved is obtained. Thesynthesized image is outputted from the image processor 6, as the outputimage 690 i.e. a wide dynamic range image (Step S5).

Second Embodiment

FIG. 13 is a functional block diagram showing a circuit arrangementprimarily relating to a noise removal processing to be executed by animage processor 6 a of a digital camera la in a second embodiment of theinvention. The image processor 6 a is substantially different from theimage processor 6 in the first embodiment in the edge componentpreservation processing. Specifically, the image processor 6 a isprovided with a high frequency generator 64′ i.e. an HPF unit 64′, anedge detecting unit 65′, and a noise removing unit 66′, and is furtherprovided with an adding section A and a subtracting section S. Thearrangements and the operations of the aforementioned parts aredescribed in detail referring to FIGS. 14 and 15. The elements of theimage processor 6 a which are substantially equivalent to those of theimage processor 6 are denoted with the same reference numerals, anddescription thereof will be omitted herein.

FIG. 14 is a partially enlarged view of the HPF unit 64′, the edgedetecting unit 65′, and the subtracting section S shown in FIG. 13. TheHPF unit 64′ includes HPF sections 641′, 642′, 643′, and 644′ to be usedin the respective processing steps 1 through n i.e. the processing steps1 through 4. As an HPF processing, the HPF sections 641′ through 644′each generates a high-frequency image by subtracting an image generatedby the edge detecting unit 65′ i.e. a below-described edge-preservedlow-frequency image, from an input image sent from a frequency divider61.

The edge detecting unit 65′ includes edge detectors 651′, 652′, 653′,and 654′ to be used in the respective processing steps 1 through 4. Theedge detectors 651′ through 654′ each performs a below-mentioned edgedetection processing with respect to the input image sent from thefrequency divider 61.

<Edge Detection Processing>

As an edge detection processing, the edge detectors 651′ through 654′each calculates an edge intensity of a low-frequency image by performingan edge intensity detection filter processing with respect to thelow-frequency image, using an edge intensity detection filter e.g. aSobel filter in each of the processing steps; and calculates an edgepreservation coefficient E based on the calculated edge intensityinformation. Then, the edge detectors 651′ through 654′ each generates alow-frequency image (hereinafter, called as an “edge-preservedlow-frequency image”) with its edge component preserved, based on theedge component preservation processing using the edge preservationcoefficient information, and outputs the edge-preserved low-frequencyimage to each of the HPF sections 641′ through 644′.

(Calculation of Edge Preservation Coefficient)

The edge intensity and the edge preservation coefficient E arecalculated as follows. Specifically, the edge detectors 651′ through654′ each calculates an edge intensity “e” by performing a filterprocessing with respect to each of the low-frequency images, using a 3×3Sobel filter i.e. by performing a 3×3 Sobel filter processing, whereinthe tap number is “3”; and calculates an edge preservation coefficient Ewith respect to the edge intensity “e” by the following formula (3).

when e<e1, E=0,

when e1<e<e2, E=(e−e1)/(e2−e1), and

when e>e2, E=1.0   (3)

where e1, e2 is a predetermined threshold value, and e1<e2.

Similarly to the first embodiment, the edge preservation threshold valuee1, e2 may be set in advance in each of the processing steps, or may beset to a common value throughout the processing steps. The informationrelating to the edge preservation threshold value e1, e2 is stored inthe edge detecting unit 65′ i.e. the edge detectors 651′ through 654′.Alternatively, a Sobel filter whose tap number is other than “3” may beused as the edge intensity detection filter, or a filter such as aPrewitt filter may be used.

(Edge Preservation Processing)

In the edge component preservation processing, the edge detectors 651′through 654′ each generates a low-frequency image i.e. theaforementioned edge-preserved low-frequency image, whose edge ispreserved with an edge preservation amount corresponding to the edgeintensity “e” by performing a weighting processing i.e. a weightedaverage processing with respect to an input image sent from an LPF unit62 to each of the edge detectors 651′ through 654′, and a low-frequencyimage, by the following formula (4), using the edge preservationcoefficient information (see FIG. 9) obtained by the (Calculation ofEdge Preservation Coefficient). By performing the edge componentpreservation processing, as shown in FIG. 9, the value of a pixel havinga larger edge preservation coefficient E i.e. closer to 1.0, isapproximated to the value of the pixel of the input image. Thereby, alarger amount of edge component is preserved. On the other hand, thevalue of a pixel having a smaller edge preservation coefficient E i.e.closer to 0, is approximated to the value of the pixel of thelow-frequency image.

Img_edge=(1.0−E)*Img_low+E*Img_input   (4)

-   where Img_edge represents an edge preserved image,-   Img_low represents a low-frequency image to be inputted to the    respective edge detectors 651′ through 654′, and-   Img_input represents an input image to be inputted to the respective    edge detectors 651′ through 654′.

Thus, the edge-preserved low-frequency images obtained by the edgedetectors 651′ through 654′ are outputted to the HPF sections 641′through 644′, respectively. The HPF sections 641′ through 644′ e.g. theHPF section 643′ generates a high-frequency image 6432′ by subtractingan edge-preserved low-frequency image 6532′ outputted from the edgedetector 653′, from an input image 6431′. Thus, theedge-component-removed high-frequency image 6432′ can be obtained bysubtracting the edge-preserved low-frequency image 6532′ from the inputimage 6431′. The edge-component-removed high-frequency image 6432′ issubjected to a noise removal processing in the following step. However,since the edge component has already been removed from thehigh-frequency image, there is no likelihood that the edge component maybe removed together with the noise component when the noise removalprocessing is executed. In other words, the noise removal processing canbe performed without removal of the edge component.

In the second embodiment, the subtracting unit S is provided downstreamin the data flow with respect to the edge detecting unit 65′. Thesubtracting unit S includes subtractors S1, S2, S3, and S4 to be used inthe respective processing steps 1 through 4. The subtractors S1 throughS4 subtract the low-frequency images outputted from the LPF sections 621through 624, from the edge-preserved low-frequency images generated bythe edge detectors 651′ through 654′, respectively, thereby extractingedge components. In this sense, the subtracting unit S serves as an edgecomponent extractor for extracting an edge component from anedge-preserved low-frequency image. Since the extracted edge componentis not subjected to a noise removal processing in the following step,the edge component is preserved without removal in a below-mentionedoutput image 690′. For instance, the subtractor S3 in the processingstep 3 obtains an edge component 6233′ by subtracting a low-frequencyimage 6232′ outputted from the LPF section 623, from the edge-preservedlow-frequency image 6532′ outputted from the edge detector 653′.

The high frequency generator 64′ uses an input image to generate ahigh-frequency image. Alternatively, it is possible to generate ahigh-frequency image, with influence of noise component residuals in aninput image 610′ being suppressed, by using an image whose degree ofsmoothing is smaller than that of an LPF to be used in the respectiveLPF sections of the frequency divider 61 e.g. by using an LPF-processedimage which has been subjected to a 3×3 LPF processing, as the inputimage 610′ to be inputted to the image processor 6 a in FIG. 13. Themodification is advantageous in prioritizing a noise removal processing,although a sharp edge in the input image 610′ may be slightly smoothedi.e. the magnitude of the edge component may be slightly decreased bythe influence of the 3×3 LPF processing, with the result that the sharpedge component may not be preserved as it is.

FIG. 15 is a partially enlarged view of the noise removing unit 66′ andthe subtracting unit A shown in FIG. 13. The noise removing unit 66′includes noise removers 661′ through 664′ to be used in the respectiveprocessing steps 1 through 4. The noise removers 661′ through 664′ eachperforms a noise removal processing substantially equivalent to the<Noise Removal Processing> described in the first embodiment, withrespect to a high-frequency image generated by the HPF unit 64′.Specifically, the noise removers 661′ through 664′ each removes a noisecomponent from the high-frequency image by performing a coringprocessing based on the coring characteristic 665 or the coringcharacteristic 666 shown in FIG. 10. For instance, the noise remover663′ in the processing step 3 obtains a noise-removed high-frequencyimage 6632′ by removing a noise component from the high-frequency image6432′ outputted from the HPF section 643′.

The adding unit A includes adders A1, A2, A3, and A4 to be used in therespective processing steps 1 through 4. The adders Al through A4 addedge component images extracted from the subtractors S1 through S4 tothe noise-removed high-frequency images obtained by the noise removers661′ through 664′, respectively. As a result of the addition,high-frequency images (hereinafter, called as “edge-preservedhigh-frequency images”) whose noise component is removed and whose edgecomponent is preserved are obtained. In this sense, the adding unit Aserves as an edge component adder for adding an edge component to ahigh-frequency image. For instance, the adder A3 in the processing step3 obtains an edge-preserved high-frequency image 6633′ by adding theedge component 6233′ outputted from the subtractor S3 to thehigh-frequency image 6632′ outputted from the noise remover 663′.

In each of the processing steps 1 through 4, in the case where an edgecomponent is attenuated, particularly in the case where an LPF-processedinput image whose degree of smoothing is small is used as the inputimage 610′ to be inputted to the image processor 6a, as mentioned above,it may be preferable to apply a predetermined gain to the edgecomponent. In light of the fact that the degree of attenuation of theedge component changes depending on the intensity of the LPF, the edgecomponent is corrected by applying a gain depending on the intensity ofthe LPF corresponding to a variation in attenuation, in other words, theedge component which gets dull resulting from smoothing by the LPFprocessing is corrected. In the modification, it is possible to providea gain setter or an edge component adjuster, as indicated by e.g. thesymbol G, between the subtracting unit S and the adding unit A in eachof the processing steps, for instance, to provide the gain setter Gbetween the subtractor S3 and the adder A3 in the processing step 3; andto cause the gain setter G to perform a processing of multiplying theedge component 6233′ with a predetermined gain. The gain setter G may beprovided in each of the processing steps in the similar manner asmentioned above.

In this way, as shown in FIG. 15, in the processing steps, thehigh-frequency images i.e. the edge-preserved high-frequency imagesoutputted from the adders A1 through A4, and a low-frequency imageoutputted from the LPF section 624 are obtained. Similarly to thedescription in the first embodiment referring to FIG. 11, a synthesizingunit 67 and a US unit 68 of a frequency synthesizing unit 69 performmultiple frequency synthesis processing and multiple US processing,respectively, with respect to the high-frequency images and thelow-frequency image in the multiple processing steps from the lowermostprocessing step 4 to the uppermost processing step 1 stepwise; andoutput the output image 690′ from a synthesizer 671, as an imagecorresponding to the input image 610′ inputted to the image processor 6a.

FIG. 16 is a flowchart showing an operation of the noise removalprocessing to be executed in the second embodiment. First, apredetermined wide dynamic range image i.e. a linear-logarithmic imageis obtained by a sensing operation or the like of an image sensor 3, andthe wide dynamic range image i.e. the input image 610′ is inputted tothe frequency divider 61 of the image processor 6 a (Step S21). Then,the frequency divider 61 i.e. the LPF unit 62 and the DS unit 63 performthe multiple LPF processing and the multiple DS processing,respectively, with respect to the input image 610′ in the multipleprocessing steps from the uppermost processing step to the lowermostprocessing step stepwise to divide the input image 610′ into pluralfrequency components, whereby a low-frequency image is generated in eachof the processing steps (Step S22).

The HPF unit 64′ generates i.e. extracts a high-frequency image, basedon the input image with respect to each of the processing steps, as thelow-frequency image is generated in each of the processing steps. Ingeneration of the high-frequency image the edge detecting unit 65′performs, as the edge detection processing, an edge intensity detectionfilter processing with respect to the low-frequency image inputted tothe edge detecting unit 65′, using an edge intensity detection filter tocalculate an edge intensity of the low-frequency image, and calculatesan edge preservation coefficient E based on the calculated edgeintensity information. Then, an edge-preserved low-frequency image whoseedge component is preserved depending on the edge intensity “e” by theedge component preservation processing is generated, by performing aweighting processing with respect to the input image inputted to theedge detecting unit 65′, and the low-frequency image, using the edgepreservation coefficient E. Then, an edge-component-removedhigh-frequency image is generated by subtracting the edge-preservedlow-frequency image from the input image (Step S23).

Then, the noise removing unit 66′ generates a noise-component-removedhigh-frequency image i.e. a coring image, by performing a noise removalprocessing, specifically, a coring processing with respect to theedge-component-removed high-frequency image, which has been inputted tothe noise removing unit 66′. Then, a high-frequency image whose noisecomponent is removed and whose edge component is preserved depending onthe edge intensity “e” is generated, by adding the edge componentobtained by subtracting the low-frequency image from the edge-preservedlow-frequency image by the subtracting unit S, to thenoise-component-removed high-frequency image, using the adding unit A(Step S24). Then, the frequency synthesizing unit 69 repeatedly performsthe frequency synthesis processing and the US processing, from thelowermost processing step to the uppermost processing step stepwise, bysumming the low-frequency image obtained in the lowermost processingstep, and the high-frequency images obtained in the respectiveprocessing steps. As a result of the summation, a synthesized imagewhose noise component is removed and whose edge component is preservedis obtained. The synthesized image is outputted from the image processor6 a, as the output image 690′ i.e. a wide dynamic range image (StepS25).

As mentioned above, in the second embodiment, the edge component ispreserved by the HPF processing by the HPF unit 64′, using theinformation obtained by the edge detecting unit 65′, whereas, in thefirst embodiment, the edge component is preserved by the noise removalprocessing by the noise removing unit 66, using the information obtainedby the edge detecting unit 65. Similarly to the first embodiment, in thesecond embodiment, the high-frequency image whose noise component isremoved and whose edge component is preserved is inputted to thefrequency synthesizing unit 69 i.e. the synthesizing unit 67. Also, bothin the first and the second embodiments, the information relating to theedge intensity and the edge preservation coefficient E concerningpreservation of the edge component is calculated by utilizing i.e.referring to the low-frequency image obtained by the LPF processing.

As mentioned above, in the first and the second embodiments, it ispossible to isolate a frequency component as a noise component from afrequency component as a real image component, which was impossible inthe conventional arrangement (see FIGS. 25 and 26). FIG. 17 is a diagramfor conceptually describing the frequency division processing to beexecuted in the first and the second embodiments. As described above, byperforming a hierarchical frequency isolation from the uppermostprocessing step 1 to the lowermost processing step n i.e. the processingstep 4, for instance, in the processing step 1, a certain imageincluding noise i. e. a wide dynamic range image is divided into ahigh-frequency component indicated by the reference numeral 701, and alow-frequency component indicated by the reference numeral 702. Then, anoise component is removed from the isolated high-frequency component701.

Then, in the processing step 2, the low-frequency component 702 isolatedin the processing step 1 is divided into a high-frequency componentindicated by the reference numeral 703, and a low-frequency componentindicated by the reference numeral 704. Thus, a processing of dividingthe low-frequency component isolated from the high-frequency componentto be removed as the noise component, into a high-frequency componentand a low-frequency component is repeated in the processing stepsthereafter stepwise. In the lowermost processing step n, thelow-frequency component isolated in the preceding step (n-1) is dividedinto a high-frequency component indicated by the reference numeral 705,and a low-frequency component indicated by the reference numeral 706.Thereby, as shown in the right portion of FIG. 17, the image is dividedinto the high-frequency components obtained in the respective processingsteps 1 through n, and the low-frequency component obtained in theprocessing step n. The low-frequency component obtained in the lowermostprocessing step n corresponds to a low-frequency image to be outputtedfrom the LPF section 624 in the processing step 4 shown in FIGS. 5 and13.

By performing the frequency division processing, as shown in FIG. 26,the high-frequency component 9222 showing that the high-frequencycomponent 9221 as a noise component is superimposed on a real imagecomponent, is divided into the high-frequency component 9221 and thehigh-frequency component 9222, as shown in FIG. 18. Thereby, theentirety of the high-frequency component 9221 of a frequency bandcorresponding to the noise component is removed by setting a noiseremoval amount 721 with such a magnitude as to cover the entirety of thehigh-frequency component 9221. On the other hand, a noise removalprocessing is performed with respect to the high-frequency component9222 of a frequency band whose noise component is small i.e. noise levelis small, by setting a noise removal amount 722 smaller than the noiseremoval amount 721 so that the removal amount of the real imagecomponent i.e. the edge component is minimized.

In the arrangement of isolating a low-frequency component in asucceeding processing step by using the low-frequency component obtainedby isolating the high-frequency component to be removed as the noisecomponent, the edge preservation coefficient is calculated based on thelow-frequency component i.e. the low-frequency image in each of theprocessing steps. This allows for performing an edge componentpreservation processing by referring to the edge preservationcoefficient calculated based on the frequency component with lesslikelihood of noise component residuals, without using thehigh-frequency component which has been generated in extracting thenoise component in each of the frequency bands i.e. in each of theprocessing steps. This enables to suppress likelihood that the edgecomponent may be erroneously detected as the noise component incalculating the edge preservation coefficient, thereby enabling tofinely preserve the edge component, as compared with the conventionalimage processing.

In the first and the second embodiments, the low-frequency image isextracted from the original image by the LPF processing, thehigh-frequency image is extracted by subtracting the low-frequency imagefrom the original image, and the noise is removed by performing a coringprocessing with respect to the high-frequency image. In the first andthe second embodiments, if the original image includes an edge where theluminance sharply changes, and the edge of the low-frequency image isprocessed by the LPF processing using a generally available linear LPF,the edge may get dull i.e. may be smoothed. On the other hand, thehigh-frequency image obtained by subtracting the low-frequency imagehaving the edge from the original image includes a frequency componentthat may serve as an edge portion corresponding to the edge of thelow-frequency image. Removing the noise component from thehigh-frequency image may result in removing the edge component. In viewof this, there are proposed the following methods (A) and (B), as amethod for preventing likelihood that an edge component may be removedwith a noise component.

(A) An LPF processing of maintaining an edge i.e. maintaining a portionwhere the luminance sharply changes is performed to prevent smoothing byLPF processing. Conventionally, as shown in FIG. 24, for instance, inthe case where the absolute value of a difference in pixel value beforeand after a sharp edge appears exceeds a predetermined threshold value εin use of an E filter as an edge preservation filter, the vicinity ofthe pixel value Xn is smoothed by performing a processing ofsubstituting the pixel value Xn at the center pixel An for the pixelvalue Xm at the pixel Am in performing an LPF processing with respect tothe center pixel An and its vicinity. Also, the vicinity of the pixelvalue Xm is smoothed in performing the LPF processing with respect tothe center pixel Am and its vicinity. Further, if the area subjected tothe LPF processing includes the pixel A representing an edge portionwhere the pixel A does not have a pixel value close to the pixel valueX, the pixel value X at the pixel A representing the edge portion isoutputted as it is. By performing the LPF processing with use of the efilter as mentioned above, the sharp change in the edge as shown by thegraphical line indicated by the reference numeral 901 is preserved. Inthe embodiments, however, the edge intensity is calculated based on theoriginal image, and the edge preservation coefficient is calculatedbased on the detected edge intensity, in place of using the ε filter sothat the sharp edge portion is preserved with the edge preservationamount corresponding to the edge intensity “e” by performing theweighting processing using the edge preservation coefficient.

(B) In this method, a coring amount i.e. a noise removal amount to beused in performing the noise removal processing is decreased at aportion where an edge exists in the original image, i.e. at a portionwhere the value of the frequency component exceeds a predeterminedvalue.

The first and the second embodiments are made based on an idea ofremoving noise by performing both of the processing described in themethods (A) and (B).

Third Embodiment

FIGS. 19 and 20 are functional block diagrams showing a circuitarrangement primarily relating to a noise removal processing to beexecuted by an image processor 6 b of a digital camera 1 b in a thirdembodiment of the invention. The image processors 6 b shown in FIGS. 19and 20 are the image processor 6 b as a subband division functioningpart, and the image processor 6 b as a subband synthesis functioningpart, respectively. In this embodiment, an input image to be inputted tothe image processor 6 b is divided into plural frequency components i.e.subbands or frequency bands by a Wavelet transformation, the frequencycomponents are subjected to a noise removal processing and an edgecomponent preservation processing, and the processed frequencycomponents are synthesized into an output image. These operations willbe described in detail in the following.

As shown in FIGS. 19 and 20, the image processor 6 b includes: a subbanddividing unit D0 comprised of a first subband divider D1, a secondsubband divider D2, . . . ; a subband synthesizing unit C0 comprised ofa first subband synthesizer C1, a second subband synthesizer C2, . . .and a noise reduction (NR) unit 89 comprised of NR sections 841, 842, .. . which are provided in correspondence to the subband synthesizers.The processing to be executed by the first subband divider D1 and thefirst subband synthesizer C1, the processing to be executed by thesecond subband divider D2 and the second subband synthesizer C2, . . .correspond to the processing step 1, the processing step 2, . . . ,respectively, in the first and the second embodiments.

The subband dividing unit D0 i.e. the first subband divider D1, thesecond subband divider D2, . . . each divides an input image IN inputtedto the image processor 6 b into plural frequency components i.e. ahigh-frequency image and a low-frequency image, using a Wavelettransformation. In other words, the subband dividing unit DO performs asubband division processing. The subband division processing isperformed by using e.g. an SSK dividing filter for Wavelettransformation. The subband dividing unit D0 includes: a horizontal HDunit 81 comprised of horizontal HD sections 801, 811, . . . which areprovided in correspondence to the first subband divider D1, the secondsubband divider D2 . . . ; a horizontal GD unit 82 comprised ofhorizontal GD sections 802, 812 . . . which are provided incorrespondence to the first subband divider D1, the second subbanddivider D2 . . .; a vertical HD unit 83 comprised of vertical HDsections 803, 813 . . . and vertical HD sections 805, 815 . . . whichare provided in correspondence to the first subband divider D1, thesecond subband divider D2 . . . ; and a vertical GD unit 84 comprised ofvertical GD sections 804, 814, . . . and vertical GD sections 806, 816,. . . which are provided in correspondence to the first subband dividerD1, the second subband divider D2 . . . .

The horizontal HD unit 81 performs a dividing filter processing in ahorizontal direction with respect to the input image IN or abelow-mentioned LL image to be generated by the vertical GD unit 84, andgenerates a high-frequency image i.e. an H-image. Specifically, as shownin FIG. 22, the horizontal HD unit 81 divides the input image IN intotwo images in the horizontal direction, and generates an H-image on theright side in the upper drawing of FIG. 22, indicated by the referencenumeral 220.

The horizontal GD unit 82 performs a dividing filter processing in thehorizontal direction with respect to the input image IN or thebelow-mentioned LL image to be generated by the vertical GD unit 84, andgenerates a low-frequency image i.e. an L-image. Specifically, as shownin FIG. 22, the horizontal HD unit 82 divides the input image IN intothe two images in the horizontal direction, and generates an L-image onthe left side in the upper drawing 220 of FIG. 22.

The vertical HD unit 83 performs a dividing filter processing in avertical direction with respect to the H-image and the L-image generatedby the horizontal HD unit 81 and the horizontal GD unit 82, andgenerates a high-frequency image in an oblique direction i.e. anHH-image, and a high-frequency image in a vertical direction i.e. anLH-image. Specifically, as shown in FIG. 22, the H-image and the L-imageare each divided into two images in the vertical direction, and theHH-image and the LH-image, which correspond to lower half portions ofthe H-image and the L-image in the lower drawing in FIG. 22, indicatedby the reference numeral 230.

The term “HD” represents a computation i.e. HD computation or HPF&1/2DScomputation, which is a subband division processing using a Wavelettransformation, and is operative to simultaneously perform HPFprocessing and one-half downsampling. The one-half downsampling isequivalent to the one-half downsampling described in the first and thesecond embodiments, but is not limited thereto. The HD computation isrealized by e.g. the following formula (5).

H(n)=(P(2n)−2P(2n+1)+P(2n+2))/2   (5)

The term “GD” represents a computation i.e. GD computation or LPF&1/2DScomputation, which is a subband division processing using a Wavelettransformation, and is operative to simultaneously perform LPFprocessing and one-half downsampling. The one-half downsampling isequivalent to the one-half downsampling described in the first and thesecond embodiments, but is not limited thereto. The GD computation isrealized by e.g. the following formula (6).

L(n)=(−P(2n−2)+2P(2n−1)+6P(2n)+2P(2n+1)−P(2n+2)/8   (6)

It should be noted that in the formulae (5) and (6), the symbol “P”represents a pixel value, and the symbols “H(n)” and “L(n)” representpixel data of the pixel number n in a high-frequency image and alow-frequency image, respectively. The symbol “(2n)” represents a pixelwhose pixel number is even, and the symbols “(2n−1)” and “(2n+1)”represent a pixel whose pixel number is odd and is different from thepixel number of the pixel (2n) by one, e.g., a pixel immediately beforeor after the pixel of the pixel number 2n.

The input image IN is divided into four images i.e. HH-image, LH-image,HL-image, and LL-image by the first subband divider D1, as shown in thelower drawing 230 in FIG. 22. Then, the LL-image among the four imagesis inputted to the second subband divider D2, and is further dividedinto four images i.e. HH-image, LH-image, HL-image, and LL-image by thesecond subband divider D2. Then, the LL-image among the four imagesdivided by the second subband divider D2 is further divided into fourimages i.e. HH-image, LH-image, HL-image, and LL-image by the thirdsubband divider D3. Thus, the subband division processing ishierarchically or stepwisely performed multiple times i.e. in themultiple processing steps in such a manner that the LL-image obtained bythe preceding processing step is divided into four images in thesucceeding processing step. In other words, octave division isperformed, wherein an image signal is divided into a low-frequencycomponent and a high-frequency component, and the division of thelow-frequency component is stepwisely repeated.

In the third embodiment, similarly to the first and the secondembodiments, the processing steps include the lowermost processing step4 i.e. the fourth subband divider D4. As far as the noise componentremoval processing and the edge component preservation processing aredesirably carried out, it is possible to provide subband dividers of alarger number or a smaller number. In an actual arrangement, the inputimage IN to be inputted to the first subband divider D1 may be 4 channelimages corresponding to colors of R, Gb, Gr, and B obtained by formatconversion in the case where an image i.e. a wide dynamic range imageacquired by the image sensor 3 is e.g. an image having a Bayerarrangement. In the modification, the acquired image is divided intofour channel images, and the processing shown in FIGS. 19 through 21 isperformed with respect to each of pixel data corresponding to the fourchannels. The image to be processed is not necessarily a color image.For instance, the aforementioned arrangement is applicable to a grayimage i.e. a monochromatic image. The same idea is also applied to thefirst and the second embodiments.

The subband synthesizing unit C0 i.e. the first subband synthesizer C1,the second subband synthesizer C2, . . . each synthesizes HH-image,LH-image, HL-image, and LL-image generated in each of the processingsteps by the subband division processing, using a Wavelettransformation. In other words, the subband synthesizing unit C0performs a subband synthesis processing. The subband synthesisprocessing is performed by using e.g. an SSK synthesizing filter forWavelet transformation. The subband synthesizing unit C0 includes: avertical PU unit 85 comprised of vertical PU sections 821 and 831, . . ., vertical PU sections 823 and 833, . . . which are provided incorrespondence to the first subband synthesizer C1, the second subbandsynthesizer C2, . . . ; a vertical QU unit 86 comprised of vertical QUsections 822 and 832, . . . , vertical QU sections 824 and 834, . . .which are provided in correspondence to the first subband synthesizerC1, the second subband synthesizer C2, . . . ; a horizontal PU unit 87comprised of horizontal PU sections 825 and 835 . . . which are providedin correspondence to the first subband synthesizer C1, the secondsubband synthesizer C2, . . . ; and a horizontal QU unit 88 comprised ofhorizontal QU sections 826 and 836, . . . which are provided incorrespondence to the first subband synthesizer C1, the second subbandsynthesizer C2.

The vertical PU unit 85 and the vertical QU unit 86 perform a synthesisfilter processing in a vertical direction with respect to the HH-image,the LH-image, the HL-image, and the LL-image generated by the verticalHD unit 83 and the vertical GD unit 84 in each of the subband dividers,and generate a high-frequency image i.e. an H-image, and a low-frequencyimage i.e. an L-image. Specifically, for instance, in the first subbandsynthesizer C1, the vertical PU section 821 and the vertical QU section822 generate an H-image by a vertical synthesis processing with respectto the HH-image and the HL-image; and the vertical PU section 823 andthe vertical QU section 824 generate an L-image by a vertical synthesisprocessing with respect to the LH-image and the LL-image. Referring toFIG. 22, the right-side H-image in the upper drawing 220 is generated bysynthesizing the HH-image and the HL-image in the lower drawing 230, andthe left-side L-image in the upper drawing 220 is generated bysynthesizing the LH-image and the LL-image in the lower drawing 230. Inthe second subband synthesizer C2, an H-image is generated bysynthesizing an HH-image 231 and an HL-image 232 in the lower drawing230, and an L-image is generated by synthesizing an LH-image 233 and anLL-image 234.

The horizontal PU unit 87 and the horizontal QU unit 88 perform asynthesis filter processing in a horizontal direction with respect tothe H-image and the L-image generated by the vertical PU unit 85 and thevertical QU unit 86, and generate an image by synthesizing the H-imageand the L-image. Specifically, for instance, in the first subbandsynthesizer C1, an output image OUT is generated by a horizontalsynthesis processing with respect to the H-image and the L-image by thehorizontal PU section 825 and the horizontal QU section 826. Referringto FIG. 22, an output image OUT is generated by synthesizing the H-imageand the L-image in the upper drawing 220. Likewise, in the secondsubband synthesizer C2, an LL-image is generated by synthesizing theH-image and the L-image by the horizontal PU section 835 and thehorizontal QU section 836. Thus, as shown in the lower drawing 230 inFIG. 22, an LL-image 235 is generated by synthesizing an image obtainedby synthesizing the HH-image 231 and the HL-image 232, and an imageobtained by synthesizing the LH-image 233 and the LL-image 234.

The term “PU” represents a computation i.e. PU computation or 2n-pixelfilter processing & 2US computation, which is a subband synthesisprocessing using a Wavelet transformation, and is operative tosimultaneously perform filter processing with respect to pixels whosepixel number is 2n (even number), and double upsampling. The doubleupsampling is equivalent to the double upsampling described in the firstand the second embodiments, but is not limited thereto. The PUcomputation is realized by e.g. the following formula (7).

P(2n)=L(n)+(H(n−1)+H(n))/4   (7)

where the symbol “P” represents a pixel value, the symbols “(n)” and“(n−1)” represent that the pixel number is n and (n−1), respectively,and the symbols “L( )” and “H( )” represent a high-frequency image and alow-frequency image, respectively. The same definition is also appliedto the below-mentioned formula (8).

The term “QU” represents a computation i.e. QU computation or(2n+1)-pixel filter processing & 2US computation, which is a subbandsynthesis processing using a Wavelet transformation, and is operative tosimultaneously perform filter processing with respect to pixels whosepixel number is (2n+1) (odd number), and double upsampling. The QUcomputation is realized by e.g. the following formula (8).

P(2n+1)=(L(n)+L(n+1))/2+(H(n−1)−6H(n)+H(n−1))/8   (8)

The PU computation and the QU computation are computations with respectto the pixels whose pixel numbers are an even number and an odd number,respectively, and the computations commonly using the terms L(n),H(n−1), and H(n), as shown in the right-hand members in the formulae (7)and (8). This is because since the upsampling (US) processing is aprocessing of generating an image with its size twice as large as thatof the original images before synthesis by synthesizing the images,images are generated from the pixels of the even number and the oddnumber, while referring to the pixel data in both of the low-frequencyimage and the high-frequency image. In the PU computation and the QUcomputation, an operation of referring to the pixel data in both of thelow-frequency image and the high-frequency image is expressed by crosslines indicated by the symbols CL1, CL2, and CL3 in FIG. 20, whichextend between the signal lines for the high-frequency image and thelow-frequency image.

The NR unit 89 performs a noise removal processing i.e. a noisereduction processing or an NR processing with respect to ahigh-frequency image to be inputted to the NR unit 89 i.e. an HH-image,an HL-image and an LH image, and a low-frequency image i.e. an LL-imagein each of the processing steps. The NR unit 89 includes the NR sections841, 842, provided in correspondence to the first subband synthesizerC1, the second subband synthesizer C2, . . . . As shown in FIG. 21, theNR sections 841, 842, . . . each includes a first coring section 891, asecond coring section 892, a third coring section 893, a coringweighting coefficient calculator 894, a first coring level setter 895, asecond coring level setter 896, a first multiplier 897, and a secondmultiplier 898.

The first coring section 891, the second coring section 892, and thethird coring section 893 perform a coring processing with respect to anHH-image, an HL-image, and an LH-image, respectively. The coringprocessing in the third embodiment is similar to the coring processingto be executed by the noise removing unit 66 in the first embodiment,wherein a noise component is removed from each of the images, using thecoring characteristic 665 or the coring characteristic 666 as shown inFIG. 10. However, the coring processing in the third embodiment isperformed depending on a coring level i.e. a coring intensity or acoring amount, which is weighed with the coring weighting coefficient tobe described later.

The coring weighting coefficient calculator 894 calculates a coringweighting coefficient. The coring weighting coefficient corresponds tothe edge preservation coefficient to be used in performing an edgecomponent preservation depending on the edge intensity, and is aweighting coefficient to be applied to the coring level so as todetermine the coring intensity or the coring degree in performing thecoring processing. Similarly to the (Calculation of Edge PreservationCoefficient) described in the first embodiment, the coring weightingcoefficient is derived by calculating an edge intensity “e” byperforming e.g. a 3×3 Sobel filter processing with respect to alow-frequency image i.e. an LL-image, and by calculating the coringweighting coefficient with respect to the calculated edge intensity “e”by the aforementioned formula (1). The thus-obtained coring weightingcoefficient is similar to the coefficient shown in FIG. 9. In otherwords, the edge preservation coefficient E corresponding to the axis ofordinate in FIG. 9 is substituted by the coring weighting coefficient.

The first coring level setter 895 and the second coring level setter 896set a coring level or a coring degree i.e. the aforementioned coringcoefficient “th” (“−th”) with respect to the high-frequency images ineach of the processing steps. The first coring level setter 895 sets acoring level with respect to an HH-image i.e. a high-frequency image,and the second coring level setter 896 sets a coring level i.e. a coringcoefficient with respect to an LH-image and an HL-image, both of whichare high-frequency images. Since the noise amount included in thehigh-frequency image differs in the processing steps, the coring levelof a different value is set in each of the processing steps.Alternatively, the first and the second coring level setters 895 and 896may be provided with a lookup table (LUT) describing the coring levelinformation with respect to each of the processing steps. Then, thecoring level corresponding to the step number information to be inputtedto the first and the second coring level setters 895 and 896 i.e.information showing a correlation between the NR section provided withthe coring level setters, and the ordinal number of the processing step,in other words, the coring level suitable for the respective processingsteps, may be set based on a data conversion processing, using the LUT.

In the third embodiment, two coring level setters i.e. the first coringlevel setter 895 with respect to an HH-image, and the second coringlevel setter 896 with respect to an HL-image and an LH-image areprovided as the coring level setting unit. Alternatively, a singlecoring level setter may be provided. Further alternatively, coring levelsetters may be provided individually with respect to an HL-image and anLH-image. In other words, three coring level setters may be provided.

The first multiplier 897 multiplies the coring level set by the firstcoring level setter 895 with the coring weighting coefficient calculatedby the coring weighting coefficient calculator 894, and outputs theinformation relating to the weighted coring level i.e. weighted coringlevel information 8971 to the first coring section 891. The secondmultiplier 898 multiplies the coring level set by the second coringlevel setter 896 with the coring weighting coefficient calculated by thecoring weighting coefficient calculator 894, and outputs the informationrelating to the weighted coring level i.e. weighted coring levelinformation 8981 to the second coring section 892 and the third coringsection 893.

Upon receiving the weighted coring level information 8971, the firstcoring section 891 adjusts the coring intensity i.e. the coring amountby changing the coring coefficient “th” (“−th”) shown in FIG. 10 basedon the weighted coring level information 8971 in performing a coringprocessing with respect to the HH-image. Similarly, the second and thethird coring sections 892 and 893 adjust the coring intensity bychanging the coring coefficient “th” (“−th”) based on the weightedcoring level information 8971 in performing a coring processing withrespect to the HL-image and the LH-image. By performing the coringprocessing, the HH-image, the HL-image, and the LH-image whose noisecomponent is removed and whose edge component is preserved depending onthe edge intensity are obtained.

The HH-image, the HL-image, and the LH-image to be inputted to each ofthe NR sections 841, 842, . . . of the NR unit 89 are the HH-image, theHL-image, and the LH-image obtained by the first subband divider D1, thesecond subband divider D2, . . . which correspond to the processingsteps 1, 2, . . . . For instance, the HH-image, the HL-image, and theLH-image to be inputted to the NR section 841 are the HH-image, theHL-image, and the LH-image to be obtained by the vertical HD section803, the vertical GD section 804, and the vertical HD section 805,respectively.

In the above arrangement, a synthesized image corresponding to the inputimage IN is restored by repeating the noise removal processing and theedge component preservation processing by the NR unit, and the subbandsynthesis processing by the subband synthesizing unit with respect tothe HH-image, the LH-image, the HL-image, and the LL-image obtained bythe subband division processing in each of the processing steps, fromthe lowermost processing step to the uppermost processing step stepwise.The synthesized image is outputted, as the output image OUT, from thefirst subband synthesizer C1 in the uppermost processing step 1 i.e.from the horizontal PU section 825.

The NR sections and the subband synthesizers are provided incorrespondence to the respective subband dividers. In the thirdembodiment, the four NR sections and the four subband synthesizers areprovided in correspondence to the first through the fourth subbanddividers D1 through D4. Also, similarly to the first and the secondembodiments, in the third embodiment, the subband synthesis processingin each of the processing steps i.e. by each of the subband synthesizersis a processing of synthesizing a high-frequency image i.e. an HH-image,an HL-image, and an LH-image outputted from the NR unit 89, and alow-frequency image i.e. an LL-image outputted from the lower-locatedprocessing step i.e. the lower-located subband synthesizer; andoutputting the synthesized image to the upper-located processing step,as a low-frequency image i.e. an LL-image, specifically, outputting thesynthesized image as the output image OUT in the processing step 1. Inthe third embodiment, a Wavelet transformation is used concerning thefrequency division processing and the frequency synthesis processing ofan image. Whereas in the first and the second embodiments, pixelinterpolation is performed in the US processing, in the thirdembodiment, the upsampling (US) processing using a Wavelettransformation is performed, specifically, the HD processing and the GDprocessing are performed in the subband division processing, and data ofrespective images generated by these processing is synthesized by the PUprocessing and the QU processing. The arrangement in the thirdembodiment is theoretically free of image degradation in performing theUS processing.

FIG. 23 is a flowchart showing an operation of the noise removalprocessing to be executed in the third embodiment. First, apredetermined wide dynamic range image i.e. a linear-logarithmic imageis obtained by a sensing operation or the like of an image sensor 3, andthe wide dynamic range image i.e. the input image IN is inputted to thesubband dividing unit D0 of the image processor 6 b i.e. the firstsubband divider D1 or the horizontal HD section 801 (Step S41). Then,the subband dividing unit D0 performs the subband division processing,i.e. the HD processing and the GD processing by Wavelet transformationwith respect to the input image IN multiple times in the multipleprocessing steps from the uppermost processing step to the lowermostprocessing step stepwise to divide the input image IN into pluralfrequency components, whereby high-frequency images i.e. an HH-image, anHL-image, and an LH-image, and a low-frequency image i.e. an LL-imageare generated in each of the processing steps (Step S42).

Then, the NR unit 89 performs a noise component removal processing andan edge component preservation processing i.e. a coring processing bythe noise component removal processing and the edge componentpreservation processing with respect to the high-frequency images i.e.the HH-image, the HL-image, and the LH-image obtained by the subbanddivision processing by the subband dividing unit D0 in each of theprocessing steps; and the subband synthesizing unit C0 repeatedlyperforms the subband synthesis processing i.e. the PU processing and theQU processing by Wavelet transformation with respect to thecoring-processed high-frequency images and the low-frequency image i.e.the LL-image, from the lowermost processing step to the uppermostprocessing step stepwise. As a result, a synthesized image whose noisecomponent is removed and whose edge component is preserved is obtained.The synthesized image is outputted from the image processor 6b, as theoutput image OUT i.e. a wide dynamic range image (Step S43).

The following is a summary of the embodiments.

As described above, in the image processing device according to thefirst through the third embodiments i.e. in the image processing device6, 6 a, 6 b, the frequency divider 61 i.e. the subband dividing sectionD0, as the frequency divider of the claimed invention, divides the inputimage 610, 610′ or the input image IN into plural frequency componentsin plural frequency bands i.e. plural processing steps; and the noiseremoving unit 66, 66′ or the first through the third coring sections 891through 893, as the noise remover of the claimed invention, removes anoise component from a high frequency component i.e. a high-frequencyimage in the frequency components in the respective frequency bandsobtained by the frequency division processing by the frequency divider.Then, the edge detecting unit 65, 65′ or the coring weightingcoefficient calculator 894, as the edge preservation informationcalculator of the claimed invention, detects the edge intensity “e”based on the low frequency component i.e. the low-frequency image in thefrequency components in the respective frequency bands obtained by thefrequency division processing by the frequency divider; and calculatesthe edge preservation information i.e. the edge preservation coefficientE relating to the degree of preserving the edge component, based on thedetected edge intensity. Then, the image processor 6, 6 a, 6 b, as theedge preserving section of the claimed invention, preserves the edgecomponent in the high frequency component, based on the edgepreservation information calculated by the edge preservation informationcalculator. The edge preserving section of the claimed inventioncorresponds to the image processor 6, 6 a, 6 b for the following reason.The image processor is functioned to control the operations of thefunctioning parts provided in the image processor e.g. the edgedetecting unit 65 or the noise removing unit 66. In view of this, theimage processor serves as the edge preserving section to control thefunctioning parts of the image processor to perform the edge componentpreservation processing.

Then, the frequency synthesizing unit 69 or the subband synthesizingunit C0, as the frequency synthesizer of the claimed invention,synthesizes the high frequency component whose noise component isremoved by the noise remover and whose edge component is preserved bythe edge preserving section, and the low frequency component, in each ofthe frequency bands. For instance, the high-frequency image 6632 and thefrequency synthesized image 6731 are synthesized, as shown in FIG. 11,or the HH-image, the HL-image, and the LH-image as the high-frequencyimage, and the LL-image as the low-frequency image are synthesized, asshown in FIG. 20. In other words, the synthesized image composed of thehigh frequency component and the low frequency component is generated ineach of the frequency bands, and the synthesized images generated in therespective frequency bands are synthesized into the output image 690,690′ or the output image OUT.

As mentioned above, the input image is divided into the frequencycomponents each having a frequency band, and the noise component isremoved from the high frequency component in the frequency componentseach having the corresponding frequency band obtained by the frequencydivision processing. In other words, since the noise component isremoved with respect to each of the frequency bands, e.g. depending onthe noise removal amount set in each of the frequency bands, the noisecomponent can be removed finely. Also, the edge intensity is detectedbased on the low frequency component in the frequency components eachhaving the corresponding frequency band, and the edge preservationinformation relating to the degree of preserving the edge component iscalculated based on the detected edge intensity. In other words, theedge intensity is detected in each of the frequency bands, and the edgepreservation information is determined based on the detected edgeintensity. Thus, the edge preservation information is determined bydetecting the edge intensity based on the low frequency with lesslikelihood of noise component residuals, in place of using the highfrequency component. This enables to preserve the edge component finely,without likelihood that the edge component may be erroneously detectedand preserved as the noise component. Thereby, even if the input imageis a wide dynamic range image, image processing with improved noiseremoval performance and improved edge preservation performance can beexecuted, thereby enabling to obtain a high-quality image.

Preferably, the frequency divider includes the LPF unit 62, as the lowfrequency generator of the claimed invention, for generating the lowfrequency component by performing an LPF processing, and the DS unit 63,as the downsampler of the claimed invention, for performing the DSprocessing with respect to the low frequency component generated by thelow frequency generator. The input image is divided into the frequencycomponents each having the corresponding frequency band by causing thelow frequency generator to perform the LPF processing, and causing thedownsampler to perform the DS processing with respect to the input imagea predetermined multiple number of times.

Preferably, the low frequency generator exclusively performs the LPFprocessing with respect to the input image in the last one of themultiple times.

In the above arrangement, the input image is divided into the frequencycomponents each having the corresponding frequency band by repeating theLPF processing and the DS processing with respect to the input image thepredetermined multiple number of times. This enables to divide the inputimage into the frequency components each having the correspondingfrequency band in a simplified construction.

Preferably, the edge preserving section i.e. the image processor 6preserves the edge component in the high frequency component in thenoise component removal processing i.e. the noise removal processing bythe noise remover i.e. the noise removing unit 66. Then, the edgepreserving section preserves the edge component in the high frequencycomponent by changing the degree of preserving the noise component inthe noise component removal processing by the noise remover, based onthe edge preservation information e.g. the edge preservation coefficientinformation 6532 calculated by the edge preservation informationcalculator i.e. the edge detecting unit 65.

In the above arrangement, the edge component in the high frequencycomponent is preserved by changing the degree of removing the noisecomponent in the noise component removal processing, based on the edgepreservation information. This enables to efficiently preserve the edgecomponent in the high frequency component by utilizing the noisecomponent removal processing, and enables to simplify the arrangement ofthe image processing device.

Preferably, the image processor 6 further includes a high frequencygenerator i.e. the HPF unit 64 for generating the high frequencycomponent from the frequency components each having the frequency bandobtained by the frequency division processing by the frequency divider.The noise remover i.e. the noise removing unit 66 performs a coringprocessing with respect to the high frequency component e.g. thehigh-frequency image 6432 generated by the high frequency generator.Then, the degree of removing the noise component in the noise componentremoval processing is changed by weight-averaging the high frequencycomponent e.g. the high-frequency image 6632 after the coring processingis performed, and the high frequency component e.g. the high-frequencyimage 6432 before the coring processing is performed, using the edgepreservation information.

In the above arrangement, the degree of removing the noise component inthe noise component removal processing is changed by weigh-averaging thehigh frequency component after the coring processing is performed, andthe high frequency component before the coring processing is performed,using the edge preservation information. This enables to realize thearrangement of changing the degree of removing the noise component in asimplified construction.

Preferably, the image processor 6 a i.e. the edge preserving sectionisolates the edge component from the high frequency component, based onthe edge preservation information calculated by the edge preservationinformation calculator i.e. the edge detecting unit 65′, and causes theadding unit A to add the edge component isolated from the high frequencycomponent, to the high frequency component whose noise component isremoved by the noise remover i.e. the noise removing unit 66′, after theedge component is isolated from the high frequency component. Thereby,the edge component in the high frequency component is preserved.

In the above arrangement, after the edge component is isolated from thehigh frequency component, the noise component is removed from the highfrequency component. This enables to remove the noise component morefinely, without likelihood that the edge component may be erroneouslydetected as the noise component in the noise component removalprocessing, and enables to preserve the edge component in the highfrequency component by synthesizing the high frequency component afterthe noise component removal, and the isolated edge component. Thisenables to more finely preserve the edge component without likelihoodthat the noise component may be erroneously detected as the edgecomponent in the edge component preservation processing.

Preferably, the image processor 6 a further includes a high frequencygenerator i.e. the HPF unit 64 for generating the high frequencycomponent from the frequency components each having the frequency bandobtained by the frequency division processing by the frequency divideri.e. the frequency divider 61. The edge preserving section i.e. theimage processor 6 a isolates the edge component from the high frequencycomponent in the high frequency generation processing by the highfrequency generator by: generating an edge-preserved low-frequencycomponent e.g. the edge-preserved low-frequency image 6532′ whose edgecomponent is preserved by weight-averaging the low frequency componente.g. the low-frequency image 6232′ generated by the LPF processing bythe low frequency generator i.e. the LPF unit 62, and the frequencycomponent before the LPF processing is performed e.g. the input image6431′, using the edge preservation information; and generating the highfrequency component e.g. the high-frequency image 6432′ by subtractingthe edge-preserved low-frequency component e.g. the edge-preservedlow-frequency component 6532′ from the frequency component before theLPF processing is performed e.g. the input image 6431′ in the highfrequency generation processing.

In the above arrangement, the edge component is isolated from the highfrequency component by generating the high frequency component bysubtracting the edge-preserved low-frequency component obtained byweight-averaging the low frequency component generated by the LPFprocessing, and the frequency component before the LPF processing isperformed, using the edge preservation information, from the frequencycomponent before the LPF processing is performed in the high frequencygeneration processing. This enables to realize the arrangement ofisolating the edge component from the high frequency component in asimplified construction, thereby allowing for easy preservation of theedge component in the high frequency component.

Preferably, the noise remover i.e. the noise removing unit 66, 66′, orthe first through the third coring sections 891 through 893 removes thenoise component by the coring processing. This enables to simplify thenoise component removal processing or the noise component removalcontrol, and to easily control the noise removal amount, using theinformation relating to a threshold value i.e. the coring coefficients“th” and “−th”.

Preferably, the frequency divider divides the input image into thefrequency components each having the frequency band by repeating, apredetermined multiple number of times, a processing of: dividing theinput image i.e. the input image IN into the high frequency componente.g. the H-image, or the HH-image, the HL-image and the LH-imageobtained by the first subband divider D1, and the low frequencycomponent e.g. the L-image or the LL-image, using a Wavelettransformation; and dividing the low frequency component e.g. theLL-image isolated from the high frequency component e.g. the HH-image,the HL-image, and the LH-image into a high frequency component e.g. theH-image, or the HH-image, the HL-image and the LH-image, and a lowfrequency component e.g. the L-image or the LL-image in the succeedingprocessing step.

In the above arrangement, the input image can be divided into thefrequency components each having the corresponding frequency band, usingthe Wavelet transformation theoretically free of image degradation.Thereby, a high-quality image can be obtained. Also, use of the Wavelettransformation is advantageous in simplifying the arrangement of theimage processor 6 b.

Preferably, the noise remover i.e. the first through the third coringsections 891 through 893, or the NR unit 89 removes the noise componentby the coring processing. The edge preserving section i.e. the imageprocessor 6 b, or the NR unit 89 preserves the edge component in thehigh frequency component i.e. the HH-image, the HL-image, and theLH-image to be inputted to the NR unit 89 shown in FIG. 21 in the coringprocessing, using the noise remover. The edge preserving sectionpreserves the edge component in the high frequency component by changingthe coring degree to be used in the coring processing, based on the edgepreservation information i.e. the edge preservation coefficient Ecalculated by the edge preservation information calculator i.e. thecoring weighting coefficient calculator 894.

In the above arrangement, the edge component in the high frequencycomponent is preserved by changing the coring degree to be used inperforming the coring processing, based on the edge preservationinformation. This enables to efficiently preserve the edge component inthe high frequency component, utilizing the coring processing.

Preferably, the noise remover i.e. the first through the third coringsections 891 through 893 changes the coring degree to be used inperforming the coring processing by changing the coring coefficientdepending on the coring level information obtained by multiplying withthe weighting coefficient i.e. the coring weighting coefficient relatingto the coring degree by the multipliers 897 and 898, based on the edgepreservation information i.e. the edge preservation coefficient E, i.e.depending on the coring level information obtained by multiplying thecoring level set by the first and the second coring level setters 895and 896, with the coring weighting coefficient.

In the above arrangement, the coring degree is changed by changing thecoring coefficient depending on the coring level information obtained bymultiplying with the weighting coefficient relating to the coring degreebased on the edge preservation information. This enables to realize thearrangement of changing the coring degree in a simplified construction.

Preferably, the noise remover i.e. the noise removing unit 66, 66′, orthe first through the third coring sections 891 through 893 is soconfigured as to change the graphical gradient “k” of the coringcoefficients “−th” through “th” of the coring characteristic 666 in thecoring processing.

In the above arrangement, it is possible to adjust the coringcharacteristic by changing the graphical gradient of the coringcoefficients of the coring characteristic in the coring processing. Thisenables to prevent likelihood that a certain pixel whose absolute valueis e.g. smaller than the coring coefficient “th” (pixels values in therange “th” through “−th”) may be removed without an exception, with theresult that a detail component relating to the texture of the image maybe removed. In other words, the arrangement enables to obtain a properimage according to needs, in which removal of a detail component in theimage is suppressed, with a negligibly small noise being allowed toremain in the image.

Preferably, the edge preservation information is an edge preservationcoefficient E which is changed as follows. If the edge intensity of thelow frequency component is larger than the predetermined threshold value“e2”, the edge preservation coefficient E is set to a predeterminedmaximal value e.g. 1.0; if the edge intensity is smaller than thepredetermined threshold value “e1” smaller than the threshold value“e2”, the edge preservation coefficient E is set to a predeterminedminimal value e.g. 0; and if the edge intensity is not smaller than thethreshold value “e1” and not larger than the threshold value “e2”, theedge preservation coefficient E is set to a value between the minimalvalue and the maximal value depending on the edge intensity.

In the above arrangement, the edge preservation coefficient is given insuch a simplified manner that if the edge intensity of the low frequencycomponent is larger than the threshold value “e2”, the edge preservationinformation is set to the predetermined maximal value; if the edgeintensity is smaller than the threshold value “e1”, the edgepreservation information is set to the predetermined minimal value; andif the edge intensity is not smaller than the threshold value “e1” andnot larger than the threshold value “e2”, the edge preservationinformation is set to any value between the minimal value and themaximal value. This makes it easy to perform the edge preservationprocessing, using the edge preservation coefficient.

Preferably, the input image is a wide dynamic range image. This enablesto perform an image processing with respect to the wide dynamic rangeimage with improved noise removal performance and improved edgepreservation performance. The same advantage is also applied to theimage processing method and the image sensing apparatus to be describedbelow.

In the image processing method according to the first through the thirdembodiment, in the frequency dividing step, an input image is dividedinto a plurality of frequency components each having a frequency band;and in a noise component removing step, a noise component is removedfrom a high frequency component in the frequency components each havingthe frequency band obtained in the frequency dividing step. Then, in anedge preservation information calculating step, an edge intensity isdetected based on a low frequency component in the frequency componentseach having the frequency band obtained in the frequency dividing step,and edge preservation information relating to a degree of preserving anedge component is calculated based on the detected edge intensity. Then,in an edge preserving step, the edge component in the high frequencycomponent is preserved, based on the edge preservation informationcalculated in the edge preservation information calculating step; and ina frequency synthesizing step, the high frequency component whose noisecomponent is removed in the noise component removing step and whose edgecomponent is preserved in the edge preserving step, and the lowfrequency component is synthesized in each of the frequency bands.

In the above arrangement, the input image is divided into the frequencycomponents each having the corresponding frequency band in the frequencydividing step, and the noise component is removed from the highfrequency component in the frequency components each having thecorresponding frequency band obtained in the frequency dividing step.Thus, since the noise component is removed with respect to each of thefrequency bands e.g. depending on the noise removal amount set in eachof the frequency bands, the noise component can be removed finely. Then,the edge intensity is detected based on the low frequency component inthe frequency components each having the corresponding frequency band,and the edge preservation information relating to the degree ofpreserving the edge component is calculated, based on the detected edgeintensity. In other words, the edge intensity is detected in each of thefrequency bands, and the edge preservation information is determinedbased on the detected edge intensity. Thus, the edge preservationinformation is determined by detecting the edge intensity based on thelow frequency with less likelihood of noise component residuals, inplace of using the high frequency component. This enables to preservethe edge component finely, without likelihood that the edge componentmay be erroneously detected and preserved as the noise component.Thereby, even if the input image is a wide dynamic range image, imageprocessing with improved noise removal performance and improved edgepreservation performance can be performed, thereby enabling to obtain ahigh-quality image.

In the image sensing apparatus according to the third through the thirdembodiments i.e. the digital camera 1, 1 a, 1 b, the image sensor 3, asthe image sensing section of the claimed invention, performs a widedynamic range imaging; a frequency divider divides a wide dynamic rangeimage acquired by the image sensing section into a plurality offrequency components each having a frequency band; and a noise removerremoves a noise component from a high frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider. Then, an edgepreservation information calculator detects an edge intensity based on alow frequency component in the frequency components each having thefrequency band obtained by the frequency division processing by thefrequency divider, and calculates edge preservation information relatingto a degree of preserving an edge component based on the detected edgeintensity. Then, an edge preserving section preserves the edge componentin the high frequency component, based on the edge preservationinformation calculated by the edge preservation information calculator,and a frequency synthesizer synthesizes the high frequency componentwhose noise component is removed by the noise remover and whose edgecomponent is preserved by the edge preserving section, and the lowfrequency component, in each of the frequency bands.

In the above arrangement, the wide dynamic range image acquired by theimage sensing section is divided into the frequency components eachhaving the corresponding frequency band, and the noise component isremoved from the high frequency component in the frequency componentseach having the corresponding frequency band obtained by the frequencydivision processing by the frequency divider. In other words, since thenoise component is removed in each of the frequency bands e.g. dependingon the noise removal amount set in each of the frequency bands, thenoise component can be removed finely. Then, the edge intensity isdetected based on the low frequency component in the frequencycomponents each having the corresponding frequency band, and the edgepreservation information relating to the degree of preserving the edgecomponent is calculated based on the detected edge intensity. In otherwords, the edge intensity is detected in each of the frequency bands,and the edge preservation information is determined based on thedetected edge intensity. Thus, the edge preservation information isdetermined by detecting the edge intensity based on the low frequencywith less likelihood of noise component residuals, in place of using thehigh frequency component. Thereby, the edge component can be preservedfinely, without likelihood that the edge component may be erroneouslydetected and preserved as the noise component. This enables to performan image processing with respect to the wide dynamic range image withimproved noise removal performance and improved edge preservationperformance, thereby enabling to obtain a high-quality image.

Preferably, the image sensing section includes the image sensor 3 whichgenerates an electric signal commensurate with an incident light amount,and which has a photoelectric conversion characteristic including alinear characteristic area where the electric signal is linearlytransformed and outputted in accordance with the incident light amount,and a logarithmic characteristic area where the electric signal islogarithmically transformed and outputted in accordance with theincident light amount.

In the above arrangement, since the image sensor 3 is capable oflogarithmically transforming and outputting the electric signal inaccordance with the incident light amount, a wide dynamic range imagecan be easily obtained by using the image sensor 3.

The image processing device according to the first through the thirdembodiments i.e. the image processor 6, 6 a, 6 b may have the followingarrangement. Specifically, the image processing device includes: afrequency divider for performing a frequency division processing ofdividing an input image into a plurality of frequency components eachhaving a frequency band; a noise remover for performing a noisecomponent removal processing of removing a noise component from a highfrequency component in the frequency components each having thefrequency band obtained by the frequency division processing by thefrequency divider; an edge preservation information calculator fordetecting an edge intensity based on a low frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider,, and calculatingedge preservation information relating to a degree of preserving an edgecomponent based on the detected edge intensity; an edge preservingsection for preserving the edge component in the high frequencycomponent, based on the edge preservation information calculated by theedge preservation information calculator; and a frequency synthesizerfor synthesizing the high frequency component whose noise component isremoved by the noise remover and whose edge component is preserved bythe edge preserving section, and a reprocessed low frequency componentobtained by performing the processing by the frequency divider, thenoise remover, the edge preservation information calculator, and theedge preserving section with respect to the low frequency componentagain, in each of the frequency bands.

Preferably, the frequency divider includes: a low frequency generatorfor generating the low frequency component by performing a low-passfilter processing; and a downsampler for performing a downsamplingprocessing with respect to the low frequency component generated by thelow frequency generator, and the frequency divider divides the inputimage into the frequency components each having the frequency band byperforming the low-pass filter processing by the low frequencygenerator, and the downsampling processing by the downsampler withrespect to the input image.

Preferably, the processing of obtaining the reprocessed low frequencycomponent is performed a predetermined multiple number of times, and thefrequency generator exclusively performs the low-pass filter processingin a last one of the multiple times.

The invention can take the following modifications.

(A) In the embodiments, a linear-logarithmic sensor having aphotoelectric conversion characteristic including a linearcharacteristic in a low luminance area, and a logarithmic orlinear-logarithmic characteristic in a high luminance area is used asthe image sensor 3 which has the different photoelectric conversioncharacteristics and is capable of performing a wide dynamic rangeimaging. Alternatively, there may be used a sensor which has a linearphotoelectric conversion characteristic both in the low luminance areaand the high luminance area i.e. a first linear characteristic and asecond linear characteristic; and has a feature that a graphicalgradient in each of the linear characteristics is changed depending onthe luminance level. Further alternatively, there may be used an imagesensor having three or more different photoelectric conversioncharacteristics, in place of the two different photoelectric conversioncharacteristics i.e. the linear characteristic and the logarithmiccharacteristic, or the first linear characteristic and the second linearcharacteristic. Further alternatively, there may be used an image sensorthat enables to obtain a wide dynamic range image by acquiring an imagein a high luminance area and an image in a low luminance area by aone-time imaging operation i.e. a one-time exposure operation, and bysynthesizing these two images. To summarize, as far as the image sensoris capable of acquiring an image having a wide dynamic range, any imagesensor can be used. In the embodiments, the image acquired by the imagesensor 3 is used as the wide dynamic range image to be inputted to theimage processor 6, 6 a, 6 b. Alternatively, any wide dynamic range imageother than the image acquired by the image sensor 3 can be used. It isneedless to say, however, that the image processor 6, 6 a, 6 b isoperative to perform an image processing with respect to an ordinaryimage other than the wide dynamic range image, in addition to theoperation of receiving the wide dynamic range image, and performing theaforementioned processing with respect to the wide dynamic range image.

(B) In the first and the second embodiments, the processing step numberand the tap number of the LPF are respectively set to the predeterminedvalues e.g. 4 (n=4) and 7 (filter size is 7×7). Alternatively, a noisedetector for detecting a noise amount in each of the processing stepsmay be provided so that the processing steps are repeated i.e. theprocessing step number is increased until the detected noise amount isequal to or smaller than a predetermined threshold value.

(C) In the embodiments, the image processor 6, 6 a, 6 b in the digitalcamera 1, 1 a, 1 b performs various processing relating to noise removalwith respect to a sensed image e.g. frequency division processing,synthesis processing, coring processing, and edge preservationprocessing. Alternatively, the various processing may be performed by aprocessor other than the digital camera. Specifically, the variousprocessing may be performed by a host processor provided with a userinterface (UI) e.g. a personal computer or a PDA (Personal DigitalAssistant), which is directly and wiredly connected to the digitalcamera 1 using e.g. a USB, or which is wirelessly connected to a networksystem by e.g. a wireless LAN, or which is so configured as to transmitinformation using a storage medium such as a memory card.

The foregoing embodiments and/or modifications primarily include theinventions having the following arrangements.

An image processing device according to an aspect comprises: a frequencydivider for performing a frequency division processing of dividing aninput image into a plurality of frequency components each having afrequency band; a noise remover for performing a noise component removalprocessing of removing a noise component from a high frequency componentin the frequency components each having the frequency band obtained bythe frequency division processing by the frequency divider; an edgepreservation information calculator for detecting an edge intensitybased on a low frequency component in the frequency components eachhaving the frequency band obtained by the frequency division processingby the frequency divider, and calculating edge preservation informationrelating to a degree of preserving an edge component based on thedetected edge intensity; an edge preserving section for preserving theedge component in the high frequency component, based on the edgepreservation information calculated by the edge preservation informationcalculator; and a frequency synthesizer for synthesizing the highfrequency component whose noise component is removed by the noiseremover and whose edge component is preserved by the edge preservingsection, and the low frequency component, in the each of the frequencybands.

Preferably, the frequency divider includes: a low frequency generatorfor generating the low frequency component by performing a low-passfilter processing; and a downsampler for performing a downsamplingprocessing with respect to the low frequency component generated by thelow frequency generator, and the input image is divided into thefrequency components in the each of the frequency bands by repeating thelow-pass filter processing by the low frequency generator, and thedownsampling processing by the downsampler with respect to the inputimage a predetermined multiple number of times.

Preferably, the edge preserving section preserves the edge component inthe high frequency component by the noise component removal processingby the noise remover by changing a degree of removing the noisecomponent in the noise component removal processing, based on the edgepreservation information calculated by the edge preservation informationcalculator.

Preferably, the image processing device further comprises: a highfrequency generator for generating the high frequency component from thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider, wherein thenoise remover performs a coring processing with respect to the highfrequency component generated by the high frequency generator, andchanges the degree of removing the noise component in the noisecomponent removal processing by weight-averaging the high frequencycomponent after the coring processing is performed and the highfrequency component before the coring processing is performed, using theedge preservation information.

Preferably, the edge preserving section isolates the edge component fromthe high frequency component as a first high frequency component, basedon the edge preservation information calculated by the edge preservationinformation calculator; and preserves the edge component in the firsthigh frequency component by synthesizing a second high frequencycomponent obtained by removing the noise component from the first highfrequency component by the noise remover, after the edge component isisolated from the first high frequency component, and the edge componentisolated from the first high frequency component.

Preferably, the image processing device further comprises a highfrequency generator for performing a high frequency generationprocessing of generating the high frequency component from the frequencycomponents each having the frequency band obtained by the frequencydivision processing by the frequency divider, wherein the edgepreserving section isolates the edge component from the high frequencycomponent generated by the high frequency generation processing by thehigh frequency generator by: generating an edge-preserved low-frequencycomponent whose edge component is preserved by weight-averaging the lowfrequency component generated by a low-pass filter processing by a lowfrequency generator, and the frequency component before the low-passfilter processing is performed, using the edge preservation information;and generating the high frequency component by subtracting theedge-preserved low-frequency component from the frequency componentbefore the low-pass filter processing is performed in the high frequencygeneration processing.

Preferably, the noise remover removes the noise component by a coringprocessing.

Preferably, the frequency divider divides the input image into thefrequency components each having the frequency band by repeating, apredetermined multiple number of times, a processing of dividing theinput image into the high frequency component and the low frequencycomponent by a Wavelet transformation, and dividing the low frequencycomponent isolated from the high frequency component into a highfrequency component and a low frequency component.

Preferably, the noise remover removes the noise component by a coringprocessing, and the edge preserving section preserves the edge componentin the high frequency component by the coring processing, using thenoise remover, and the edge preserving section preserves the edgecomponent in the high frequency component by changing a coring degree inthe coring processing, based on the edge preservation informationcalculated by the edge preservation information calculator.

Preferably, the noise remover changes the coring degree by changing acoring coefficient depending on coring level information obtained bymultiplying with a weighting coefficient relating to the coring degreebased on the edge preservation information.

Preferably, the noise remover is so configured as to change a graphicalgradient of coring coefficients of a coring characteristic in the coringprocessing.

Preferably, the edge preservation information is an edge preservationcoefficient which is changed as follows: if the edge intensity of thelow frequency component is larger than a predetermined first thresholdvalue, the edge preservation coefficient is set to a predetermined fixedmaximal value; if the edge intensity is smaller than a predeterminedsecond threshold value smaller than the first threshold value, the edgepreservation coefficient is set to a predetermined fixed minimal value;and if the edge intensity is not smaller than the second threshold valueand not larger than the first threshold value, the edge preservationcoefficient is set to a value between the minimal value and the maximalvalue depending on the edge intensity.

Preferably, the input image is a wide dynamic range image.

An image processing method according to another aspect of the inventioncomprises: a frequency dividing step of dividing an input image into aplurality of frequency components each having a frequency band; a noisecomponent removing step of removing a noise component from a highfrequency component in the frequency components each having thefrequency band obtained in the frequency dividing step; an edgepreservation information calculating step of detecting an edge intensitybased on a low frequency component in the frequency components eachhaving the frequency band obtained in the frequency dividing step, andcalculating edge preservation information relating to a degree ofpreserving an edge component based on the detected edge intensity; anedge preserving step of preserving the edge component in the highfrequency component, based on the edge preservation informationcalculated in the edge preservation information calculating step; and afrequency synthesizing step of synthesizing the high frequency componentwhose noise component is removed in the noise component removing stepand whose edge component is preserved in the edge preserving step, andthe low frequency component, in the each of the frequency bands.

Preferably, the input image is a wide dynamic range image.

An image sensing apparatus according to yet another aspect of theinvention comprises: an image sensing section so configured as toperform a wide dynamic range imaging; a frequency divider for performinga frequency division processing of dividing a wide dynamic range imageacquired by the image sensing section into a plurality of frequencycomponents in a plurality of frequency bands; a noise remover forremoving a noise component from a high frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider; an edgepreservation information calculator for detecting an edge intensitybased on a low frequency component in the frequency components eachhaving the frequency band obtained by the frequency division processingby the frequency divider, and calculating edge preservation informationrelating to a degree of preserving an edge component based on thedetected edge intensity; an edge preserving section for preserving theedge component in the high frequency component, based on the edgepreservation information calculated by the edge preservation informationcalculator; and a frequency synthesizer for synthesizing the highfrequency component whose noise component is removed by the noiseremover and whose edge component is preserved by the edge preservingsection, and the low frequency component, in the each of the frequencybands.

Preferably, the image sensing section includes an image sensor whichgenerates an electric signal commensurate with an incident light amount,and which has a photoelectric conversion characteristic including alinear characteristic area where the electric signal is linearlytransformed and outputted in accordance with the incident light amount,and a logarithmic characteristic area where the electric signal islogarithmically transformed and outputted in accordance with theincident light amount.

Preferably, the frequency generator exclusively performs the low-passfilter processing with respect to the input image in a last one of themultiple times.

An image processing device according to still another aspect of theinvention comprises: a frequency divider for performing a frequencydivision processing of dividing an input image into a plurality offrequency components each having a frequency band; a noise remover forperforming a noise component removal processing of removing a noisecomponent from a high frequency component in the frequency componentseach having the frequency band obtained by the frequency divisionprocessing by the frequency divider; an edge preservation informationcalculator for detecting an edge intensity based on a low frequencycomponent in the frequency components each having the frequency bandobtained by the frequency division processing by the frequency divider,and calculating edge preservation information relating to a degree ofpreserving an edge component based on the detected edge intensity; anedge preserving section for preserving the edge component in the highfrequency component, based on the edge preservation informationcalculated by the edge preservation information calculator; and afrequency synthesizer for synthesizing the high frequency componentwhose noise component is removed by the noise remover and whose edgecomponent is preserved by the edge preserving section, and a reprocessedlow frequency component obtained by performing the processing by thefrequency divider, the noise remover, the edge preservation informationcalculator, and the edge preserving section with respect to the lowfrequency component again, in the each of the frequency bands.

Preferably, the frequency divider includes: a low frequency generatorfor generating the low frequency component by performing a low-passfilter processing; and a downsampler for performing a downsamplingprocessing with respect to the low frequency component generated by thelow frequency generator, and the frequency divider divides the inputimage into the frequency components each having the frequency band byperforming the low-pass filter processing by the low frequencygenerator, and the downsampling processing by the downsampler withrespect to the input image.

Preferably, the processing of obtaining the reprocessed low frequencycomponent is performed a predetermined multiple number of times, and thelow frequency generator exclusively performs the low-pass filterprocessing in a last one of the multiple times.

Although the present invention has been fully described by way ofexample with reference to the accompanying drawings, it is to beunderstood that various changes and modifications will be apparent tothose skilled in the art. Therefore, unless otherwise such changes andmodifications depart from the scope of the present invention hereinafterdefined, they should be construed as being included therein.

1. An image processing device, comprising: a frequency divider forperforming a frequency division processing of dividing an input imageinto a plurality of frequency components each having a frequency band; anoise remover for performing a noise component removal processing ofremoving a noise component from a high frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider; an edgepreservation information calculator for detecting an edge intensitybased on a low frequency component in the frequency components eachhaving the frequency band obtained by the frequency division processingby the frequency divider, and calculating edge preservation informationrelating to a degree of preserving an edge component based on thedetected edge intensity; an edge preserving section for preserving theedge component in the high frequency component, based on the edgepreservation information calculated by the edge preservation informationcalculator; and a frequency synthesizer for synthesizing the highfrequency component whose noise component is removed by the noiseremover and whose edge component is preserved by the edge preservingsection, and the low frequency component, in the each of the frequencybands.
 2. The image processing device according to claim 1, wherein thefrequency divider includes: a low frequency generator for generating thelow frequency component by performing a low-pass filter processing; anda downsampler for performing a downsampling processing with respect tothe low frequency component generated by the low frequency generator,and the input image is divided into the frequency components in the eachof the frequency bands by repeating the low-pass filter processing bythe low frequency generator, and the downsampling processing by thedownsampler with respect to the input image a predetermined multiplenumber of times.
 3. The image processing device according to claim 1,wherein the edge preserving section preserves the edge component in thehigh frequency component by the noise component removal processing bythe noise remover by changing a degree of removing the noise componentin the noise component removal processing, based on the edgepreservation information calculated by the edge preservation informationcalculator.
 4. The image processing device according to claim 3, furthercomprising: a high frequency generator for generating the high frequencycomponent from the frequency components each having the frequency bandobtained by the frequency division processing by the frequency divider,wherein the noise remover performs a coring processing with respect tothe high frequency component generated by the high frequency generator,and changes the degree of removing the noise component in the noisecomponent removal processing by weight-averaging the high frequencycomponent after the coring processing is performed and the highfrequency component before the coring processing is performed, using theedge preservation information.
 5. The image processing device accordingto claim 1, wherein the edge preserving section isolates the edgecomponent from the high frequency component as a first high frequencycomponent, based on the edge preservation information calculated by theedge preservation information calculator; and preserves the edgecomponent in the first high frequency component by synthesizing a secondhigh frequency component obtained by removing the noise component fromthe first high frequency component by the noise remover, after the edgecomponent is isolated from the first high frequency component, and theedge component isolated from the first high frequency component.
 6. Theimage processing device according to claim 5, further comprising a highfrequency generator for performing a high frequency generationprocessing of generating the high frequency component from the frequencycomponents each having the frequency band obtained by the frequencydivision processing by the frequency divider, wherein the edgepreserving section isolates the edge component from the high frequencycomponent generated by the high frequency generation processing by thehigh frequency generator by: generating an edge-preserved low-frequencycomponent whose edge component is preserved by weight-averaging the lowfrequency component generated by a low-pass filter processing by a lowfrequency generator, and the frequency component before the low-passfilter processing is performed, using the edge preservation information;and generating the high frequency component by subtracting theedge-preserved low-frequency component from the frequency componentbefore the low-pass filter processing is performed in the high frequencygeneration processing.
 7. The image processing device according to claim5, wherein the noise remover removes the noise component by a coringprocessing.
 8. The image processing device according to claim 1, whereinthe frequency divider divides the input image into the frequencycomponents each having the frequency band by repeating, a predeterminedmultiple number of times, a processing of dividing the input image intothe high frequency component and the low frequency component by aWavelet transformation, and dividing the low frequency componentisolated from the high frequency component into a high frequencycomponent and a low frequency component.
 9. The image processing deviceaccording to claim 8, wherein the noise remover removes the noisecomponent by a coring processing, and the edge preserving sectionpreserves the edge component in the high frequency component by thecoring processing, using the noise remover, and the edge preservingsection preserves the edge component in the high frequency component bychanging a coring degree in the coring processing, based on the edgepreservation information calculated by the edge preservation informationcalculator.
 10. The image processing device according to claim 9,wherein the noise remover changes the coring degree by changing a coringcoefficient depending on coring level information obtained bymultiplying with a weighting coefficient relating to the coring degreebased on the edge preservation information.
 11. The image processingdevice according to claim 4, wherein the noise remover is so configuredas to change a graphical gradient of coring coefficients of a coringcharacteristic in the coring processing.
 12. The image processing deviceaccording to claim 1, wherein the edge preservation information is anedge preservation coefficient which is changed as follows: if the edgeintensity of the low frequency component is larger than a predeterminedfirst threshold value, the edge preservation coefficient is set to apredetermined fixed maximal value; if the edge intensity is smaller thana predetermined second threshold value smaller than the first thresholdvalue, the edge preservation coefficient is set to a predetermined fixedminimal value; and if the edge intensity is not smaller than the secondthreshold value and not larger than the first threshold value, the edgepreservation coefficient is set to a value between the minimal value andthe maximal value depending on the edge intensity.
 13. The imageprocessing device according to claim 1, wherein the input image is awide dynamic range image.
 14. An image processing method, comprising: afrequency dividing step of dividing an input image into a plurality offrequency components each having a frequency band; a noise componentremoving step of removing a noise component from a high frequencycomponent in the frequency components each having the frequency bandobtained in the frequency dividing step; an edge preservationinformation calculating step of detecting an edge intensity based on alow frequency component in the frequency components each having thefrequency band obtained in the frequency dividing step, and calculatingedge preservation information relating to a degree of preserving an edgecomponent based on the detected edge intensity; an edge preserving stepof preserving the edge component in the high frequency component, basedon the edge preservation information calculated in the edge preservationinformation calculating step; and a frequency synthesizing step ofsynthesizing the high frequency component whose noise component isremoved in the noise component removing step and whose edge component ispreserved in the edge preserving step, and the low frequency component,in the each of the frequency bands.
 15. The image processing methodaccording to claim 14, wherein the input image is a wide dynamic rangeimage.
 16. An image sensing apparatus, comprising: an image sensingsection so configured as to perform a wide dynamic range imaging; afrequency divider for performing a frequency division processing ofdividing a wide dynamic range image acquired by the image sensingsection into a plurality of frequency components in a plurality offrequency bands; a noise remover for removing a noise component from ahigh frequency component in the frequency components each having thefrequency band obtained by the frequency division processing by thefrequency divider; an edge preservation information calculator fordetecting an edge intensity based on a low frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider, and calculatingedge preservation information relating to a degree of preserving an edgecomponent based on the detected edge intensity; an edge preservingsection for preserving the edge component in the high frequencycomponent, based on the edge preservation information calculated by theedge preservation information calculator; and a frequency synthesizerfor synthesizing the high frequency component whose noise component isremoved by the noise remover and whose edge component is preserved bythe edge preserving section, and the low frequency component, in theeach of the frequency bands.
 17. The image sensing apparatus accordingto claim 16, wherein the image sensing section includes an image sensorwhich generates an electric signal commensurate with an incident lightamount, and which has a photoelectric conversion characteristicincluding a linear characteristic area where the electric signal islinearly transformed and outputted in accordance with the incident lightamount, and a logarithmic characteristic area where the electric signalis logarithmically transformed and outputted in accordance with theincident light amount.
 18. The image processing device according toclaim 2, wherein the frequency generator exclusively performs thelow-pass filter processing with respect to the input image in a last oneof the multiple times.
 19. An image processing device, comprising: afrequency divider for performing a frequency division processing ofdividing an input image into a plurality of frequency components eachhaving a frequency band; a noise remover for performing a noisecomponent removal processing of removing a noise component from a highfrequency component in the frequency components each having thefrequency band obtained by the frequency division processing by thefrequency divider; an edge preservation information calculator fordetecting an edge intensity based on a low frequency component in thefrequency components each having the frequency band obtained by thefrequency division processing by the frequency divider, and calculatingedge preservation information relating to a degree of preserving an edgecomponent based on the detected edge intensity; an edge preservingsection for preserving the edge component in the high frequencycomponent, based on the edge preservation information calculated by theedge preservation information calculator; and a frequency synthesizerfor synthesizing the high frequency component whose noise component isremoved by the noise remover and whose edge component is preserved bythe edge preserving section, and a reprocessed low frequency componentobtained by performing the processing by the frequency divider, thenoise remover, the edge preservation information calculator, and theedge preserving section with respect to the low frequency componentagain, in the each of the frequency bands.
 20. The image processingdevice according to claim 19, wherein the frequency divider includes: alow frequency generator for generating the low frequency component byperforming a low-pass filter processing; and a downsampler forperforming a downsampling processing with respect to the low frequencycomponent generated by the low frequency generator, and the frequencydivider divides the input image into the frequency components eachhaving the frequency band by performing the low-pass filter processingby the low frequency generator, and the downsampling processing by thedownsampler with respect to the input image.
 21. The image processingdevice according to claim 20, wherein the processing of obtaining thereprocessed low frequency component is performed a predeterminedmultiple number of times, and the low frequency generator exclusivelyperforms the low-pass filter processing in a last one of the multipletimes.